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INTRODUCTION 


PIRLS 2021—20 Years of Monitoring Trends 
in International Reading Achievement 


Ina V.S. Mullis 


IEAs PIRLS (Progress in International Reading Literacy Study) was inaugurated in 2001 as a 
follow-up to IEA’ 1991 Reading Literacy Study. Conducted every five years, PIRLS assesses 
international trends in the reading comprehension of young students in their fourth year of 
schooling—an important transition point in children’s development as readers. Typically, at this 
point in their schooling, students have learned how to read and are now reading to learn. PIRLS was 
designed to complement IEA’s TIMSS assessments of mathematics and science at the fourth grade. 
PIRLS is directed by IEA’s TIMSS & PIRLS International Study Center at Boston College, working in 
close cooperation with IEA Amsterdam and IEA Hamburg. 

Because developing reading literacy is vital to every student’s growth, education, and daily 
life, IEA (the International Association for the Evaluation of Educational Achievement) has been 
conducting regular international assessments of reading achievement and the contexts for learning 
to read for almost 60 years. IEA is an independent international cooperative of national research 
institutions and government agencies that pioneered international comparative assessments of 
educational achievement in the 1960s to gain a deeper understanding of policy effects across 
countries’ different systems of education. Each successive PIRLS assessment has continued in 
this tradition, consisting of a state-of-the-art reading assessment measuring trends in reading 
achievement and accompanied by an extensive set of questionnaires for parents or caregivers, schools, 
teachers, countries, and the students themselves. The questionnaire data about students’ contexts for 
learning to read and educational opportunities provide important information for interpreting the 
achievement results. 

The first two chapters of the PIRLS 2021 Assessment Frameworks contain the PIRLS 2021 
Reading Framework and PIRLS 2021 Context Questionnaire Framework, respectively. The reading 
framework provides guidelines for assessing reading comprehension at the fourth grade according 
to a matrix of two reading purposes—literary and informational—by four comprehension strategies 
(retrieval, inferencing, integrating, and evaluation). The context questionnaire framework describes 
the topics to be covered by the PIRLS 2021 questionnaires. The third chapter describes the 
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assessment design for PIRLS 2021. Across the past 20 years, linking the PIRLS assessments of reading 
achievement to the rich array of PIRLS questionnaire data about the contexts in which students learn 
to read has provided researchers an important source of policy relevant information about how to 
improve reading education around the world. 


History of PIRLS 2021 


PIRLS 2021 is the fifth assessment in the current trend series, following PIRLS 2001, 2006, 2011, and 
2016. The number of countries participating in PIRLS has grown with each assessment. Nearly 70 
countries and sub-national benchmarking entities are participating in PIRLS 2021, including many 
that have participated in previous cycles since 2001. For countries with data since 2001, PIRLS 2021 
will provide the fifth in a series of trend achievement measures collected over 20 years. 

In 2001, countries that had participated in IEA’s reading literacy assessments wanted to 
work with IEA and Boston College to build a new innovative reading assessment. This included a 
commitment to extend the information PIRLS collects about student educational contexts for learning 
to read. Since 2001, PIRLS has included the Learning to Read Survey completed by students’ parents 
or caregivers as central to the questionnaires. There also is a PIRLS Encyclopedia produced as part of 
each assessment cycle, which contains comparative system-level information across countries and a 
chapter written by each participating country describing its own reading curriculum and instruction. 

Since its creation in 2001, PIRLS has been a collaborative effort among the participating 
countries and IEAs TIMSS & PIRLS International Study Center. All the countries, institutions, and 
agencies involved in successive PIRLS assessments have worked to improve PIRLS and build the 
most comprehensive and innovative measure of reading comprehension possible. In 2006, PIRLS was 
expanded from 8 to 10 passages to enable reporting results by reading comprehension processes in 
addition to literary and informational purposes. 

In 2011, the TIMSS and PIRLS assessment cycles came together, providing a unique opportunity 
for countries to collect reading, mathematics, and science achievement data on the same fourth 
grade students. Particular effort was expended on updating the questionnaires and coordinating 
them across PIRLS and TIMSS. Also, in 2011 IEA broadened the PIRLS assessment coverage to meet 
the needs of countries in which most children in the fourth grade are still developing fundamental 
reading skills. 

PIRLS 2016 was further increased to 20 passages to include a second assessment option—PIRLS 
Literacy, a less difficult reading assessment which was equivalent in scope to PIRLS. Also, ePIRLS— 
an assessment of online reading—was introduced in 2016 as another option. ePIRLS addresses the 
ever increasing prevalence of online reading. The internet often is the primary way students acquire 
information and the central source for students to conduct research in their school subjects. ePIRLS 
uses an engaging simulated internet environment to present fourth grade students with school-like 
assignments involving science and social studies topics. 
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PIRLS 2021—digitalPIRLS Transitioning to the Future 


Consistent with the drive to innovate with each successive PIRLS cycle, PIRLS 2021 is transitioning 
from paper-based booklets to a digital environment. About half the countries will deliver PIRLS 2021 
via computers, using a streamlined, easy-to-use user interface that allows students to manage reading 
the passages and answering the questions together in one seamless process. The colorful passages are 
designed to be engaging, and there are new item types to facilitate computerized scoring. 

digitalPIRLS 2021 is administered through an eAssessment system that brings efficiency to the 
operational aspects of PIRLS, including computerized delivery of assessment materials to students 
(no more printing and distributing booklets). This enables ePIRLS to be integrated with digitalPIRLS 
so that ePIRLS does not require an additional day of assessment. 

digitalPIRLS 2021 also provides some scope to adjust the assessment design. The computerized 
digitalPIRLS 2021 can integrate the PIRLS passages and the less difficult PIRLS Literacy passages 
in flexible ways, making it possible to target the difficulty of the PIRLS 2021 assessment to the level 
of achievement of the student population in the participating countries. By capitalizing on the wide 
range in difficulty of the passages developed for PIRLS and PIRLS Literacy in 2016 and continuing 
to expand the difficulty range with the newly developed reading passages, one unified PIRLS 2021 
assessment can better measure the range of high, medium, and low reading abilities found in each of 
the PIRLS participating countries. 


Updating the PIRLS Frameworks for PIRLS 2021 


Updating the PIRLS frameworks with each assessment cycle provides participating countries 

with opportunities to introduce fresh ideas and current information about curricula, standards, 
frameworks, and instruction. This keeps the frameworks educationally relevant, creates coherence 
from assessment to assessment, and permits the frameworks, instruments, and procedures to evolve 
gradually into the future. 

For PIRLS 2021, the frameworks were updated using information provided through reviews by 
the National Research Coordinators (NRCs) from the participating countries and the descriptions 
of curriculum and instruction described in the PIRLS 2021 Encyclopedia. The PIRLS 2021 expert 
committees, the Reading Development Group (RDG) and the Questionnaire Development Group 
(QDG), also provided very important ideas and direction. There was an iterative process of the 
frameworks being reviewed and revised by the NRCs and expert committees prior to publication. 
IEAs TIMSS & PIRLS International Study Center gratefully acknowledges the many important 
contributions made throughout the process. 
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CHAPTER 1 


PIRLS 2021 Reading Assessment Framework 


Ina V.S. Mullis and Michael O. Martin 


Overview 


In 2021, IEA’s PIRLS (Progress in International Reading Literacy Study) conducts its fifth reading 
assessment, providing data on 20 years of trends in comparative reading achievement across 
countries. Reading literacy is the foundation for student academic success and personal growth, and 
PIRLS is a valuable vehicle for studying whether new or revised policies impact achievement. The 
PIRLS 2021 Reading Assessment Framework and the instruments developed to assess this framework 
reflect IEA’s commitment to be forward thinking. 

For 2021, PIRLS is focusing on converting to a digital format. Presenting PIRLS reading passages 
and items via computer will deliver an engaging and visually attractive experience that will motivate 
students and increase operational efficiency. Also, PIRLS 2021 can be administered in the same 
digitally based environment as ePIRLS 2021, the computer-based assessment of online reading in a 
simulated internet environment that was initiated in 2016. 

PIRLS is based on a broad notion of what the ability to read means—a notion that includes 
reading for the pleasure it provides in allowing us to experience different worlds, other cultures, and 
a host of new ideas. It also encompasses reflecting on written texts and other sources of information 
as tools for attaining individual and societal goals, also known as “reading to do”! This view is 
increasingly relevant in today’s society, where greater emphasis continues to be placed on students’ 
ability to use the information they gain from reading.?>+ Emphasis is shifting from demonstrating 
fluency and basic comprehension to demonstrating the ability to apply what is understood or 
comprehended to new situations or projects, see also PIRLS 2016 Encyclopedia.>*” 


The PIRLS framework for assessing reading achievement was initially developed for the first 
assessment in 2001, using IEA’s 1991 Reading Literacy Study**!” as the basis for the PIRLS definition 
of reading literacy and for establishing the aspects of reading comprehension to be assessed. 

Since then, the PIRLS assessment framework has been updated for each subsequent assessment 
cycle!!213.14 and now for PIRLS 2021. 
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A Definition of Reading Literacy 


The PIRLS definition of reading literacy is grounded in IEAs 1991 study, in which reading literacy 
was defined as “the ability to understand and use those written language forms required by society 
and/or valued by the individual”. 

With successive assessments, this definition has been elaborated so that it retains its applicability 
to readers of all ages and a broad range of written language forms, yet makes explicit reference to 
aspects of the reading experience of young students as they become proficient readers, highlights 
the widespread importance of reading in school and everyday life, and acknowledges the increasing 
variety of texts in today’s technological world. Currently, the PIRLS definition of reading literacy is as 
follows: 


Reading literacy is the ability to understand and use those written language forms 
required by society and/or valued by the individual. Readers can construct meaning 
from texts in a variety of forms. They read to learn, to participate in communities of 
readers in school and everyday life, and for enjoyment. 


This view of reading reflects numerous theories of reading 


Throughout the framework, literacy as a constructive and interactive process,}©!71819,20,21,22 


various sources that have Meaning is constructed through the interaction between reader and 


providedialresearchiand text in the context of a particular reading experience.***4 Readers 


SAMO TE BEMS NaN: are regarded as actively constructing meaning, reasoning with the 

framework are referenced. ; : . ' 

text, and knowing effective reading strategies and how to reflect on 

These references represent ; 
y reading.» 

the volumes of literature 


Before, during, and after reading, readers use a repertoire 
and research that have 


avxe)daakexemdatem eal ni ies) 


licclantennelaemlarel [ure liare) 


considerable research by 


of linguistic skills, cognitive and metacognitive strategies, as well 
as background knowledge to construct meaning .?77829.30.3132 [ny 
addition, the context of the reading situation can support the 


countries participating in construction of meaning by promoting engagement and motivation 


PIRLS. to read, but the context also can place specific demands that might 


not support the construction of meaning.*3°4956 

In order to acquire knowledge of the world and themselves, readers can learn from a host of 
text types. Each text type follows conventional forms and rules which aid the reader's interpretation 
of the text.” 

Any given text type can take many forms and combinations of forms. These include traditional 
written forms, such as books, magazines, documents, and newspapers, as well as digital forms that 
include the numerous ways of communicating via the internet and websites where text often is 
integrated with various multimedia formats.**°94041 
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Increasingly, internet reading is a key component of school curricula and one of the central ways 
students acquire information.*?344 New digital literacies are necessary for reading on the internet, 
where a successful reader is one that can meet his or her reading goals by efficiently finding and 
comprehending the target information.45464748 

The internet is a nonlinear network of texts distributed across multiple websites and pages. 
Looking for and learning information from the internet involves comprehension of information 
arranged within this complex reading environment.°°°!»?°3 While traditional printed text usually 
is read in a linear fashion, online reading consists of searching through a network of multiple texts 
where readers are responsible for creating their own paths. Readers first must access the appropriate 
website, and then use navigation strategies (e.g., multiple navigation and sub-navigation menus, tabs, 
and links) to move efficiently within and across one webpage or site to the next. 

Essentially, reading for informational purposes on the internet requires all of the reading 
comprehension skills and strategies necessary for reading traditional printed text, but in a different 
environment containing much more information.*4 Because of the complexity of the internet, online 
reading involves being able to use reading comprehension skills and strategies in contexts that are 
very different from those encountered in reading traditional printed materials.* 

Whether reading online or printed text, discussing what they have read with different groups 
of individuals allows young students to construct text meaning in a variety of contexts.>*°” Social 
interactions about reading in one or more communities of readers can be instrumental in helping 
young students gain an understanding and appreciation of texts and other sources of information.***? 
Socially constructed environments in the classroom or school library can give young students formal 
and informal opportunities to broaden their perspectives and see reading as a shared experience with 
their classmates and teachers.®*! This can be extended to communities outside of school as young 
students talk with their families and friends about ideas and information acquired from reading. 


The PIRLS Framework for Assessing Reading Achievement 


Based on reading purposes and comprehension processes, the PIRLS 2021 framework provides the 
foundation for the PIRLS international assessments of students’ reading achievement in their fourth 
year of schooling. 


e PIRLS, now in its 20th year is well-established as the “de facto” worldwide standard for reading 
comprehension achievement at primary school level. PIRLS 2021 is transitioning to a digital 
format (but also will be offered in the traditional paper-and-pencil format). 


e PIRLS 2021 includes passages that range in difficulty, but the assessment design enables the 
results to be reported on the same achievement scale. 


e ePIRLS computer-based tasks extend PIRLS to assess how well students read, interpret, and 
critique online information in an environment that looks and feels like the internet. 
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As shown in Exhibit 1, the PIRLS framework focuses on the two overarching purposes for 
reading that account for most of the reading done by young students both in and out of school: for 
literary experience, and to acquire and use information. In addition, the PIRLS assessment integrates 
four broad-based comprehension processes within each of the two purposes for reading: focus on and 
retrieve explicitly stated information, make straightforward inferences, interpret and integrate ideas 
and information, and evaluate and critique content and textual elements. It should be acknowledged 
that the purposes for reading and the processes of comprehension do not function in isolation from 
one another or from the context in which students live and learn. 


Exhibit 1: The PIRLS Reading Purposes and Comprehension Processes 


Purposes for Reading 


Literary Experience 

Acquire and Use Information 

Focus on and Retrieve Explicitly Stated Information 
Make Straightforward Inferences 

Interpret and Integrate Ideas and Information 


Evaluate and Critique Content and Textual Elements 


PIRLS Framework Emphases in PIRLS and ePIRLS 


The two reading purposes and four comprehension processes form the basis for assessing PIRLS as 
well as ePIRLS online reading. Exhibit 2 presents the reading purposes and processes assessed by 
PIRLS and the percentages of the assessment devoted to each. 


Exhibit 2: Percentages of the PIRLS and ePIRLS Reading Assessments Devoted to Each Reading Purpose 
and Comprehension Process 


PIRLS ePIRLS 


Purposes for Reading 
Literary Experience 50% 0% 
Acquire and Use Information 50% 100% 


Processes of Comprehension 


Focus on and Retrieve Explicitly Stated Information 20% 20% 
Make Straightforward Inferences 30% 30% 
Interpret and Integrate Ideas and Information 30% 30% 
Evaluate and Critique Content and Textual Elements 20% 20% 
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Purposes for Reading 


Throughout the world, reading literacy is directly related to the reasons people read; broadly, these 
reasons include reading for pleasure and personal interest, learning, and participation in society. The 
early reading of most young students often includes reading of narrative texts that tell a story (e.g., 
storybooks or picture books) or informational texts that tell students about the world around them 
and answer questions. As young students develop their literacy abilities and are increasingly required 
to read in order to learn across the curriculum, reading to acquire information from books and other 
print materials becomes more important.®64% 

Aligned with these reading purposes, PIRLS assessments focus on reading for literary experience 
and reading to acquire and use information. Because both purposes for reading are important for 
young students, PIRLS contains an equal proportion of material assessing each purpose. However, 
because much online reading is done for the purpose of acquiring information, the ePIRLS tasks 
specifically focus on reading to acquire and use information. 

The ePIRLS assessment tasks assess reading for information. The tasks simulate websites from 
the internet from which students gather information, using links and tabs to navigate through texts 
and graphics, to accomplish school-based research projects. The approach is based on using websites 
from the actual internet as the basis for creating a closed internet environment, through which fourth 
grade students can accomplish an online study of a science or social studies topic, similar to the types 
of projects or reports they might be asked to complete for school. 

The PIRLS passages are classified by their primary purposes, and the accompanying questions 
address these purposes for reading. That is, passages classified as literary have questions addressing 
theme, plot events, characters, and setting, and those classified as informational are accompanied 
by questions about the information contained in the passages. Although the passages distinguish 
between purposes for reading, the comprehension processes readers use are more similar than 
different for both purposes; therefore, the comprehension processes are evaluated across all passages, 
including the ePIRLS internet-like tasks. 

Each purpose for reading often is associated with certain types of texts. For example, reading 
for literary experience often is accomplished through reading fiction, while reading to acquire and 
use information generally is associated with informative articles and instructional texts. However, the 
purposes for reading do not align strictly with text types. For example, biographies or autobiographies 
can be primarily informational or literary, but include characteristics of both purposes. 

Texts often differ in the way in which ideas are organized and presented, eliciting a variety of 
ways to construct meaning.®° Text organization and format can vary to a great degree, ranging from 
sequential ordering of written material to snippets of words and phrases arranged with pictorial and 
tabular data. The content, organization, and style that may be typical of a particular text genre have 
implications for the reader’s approach to understanding the text.9%6)70.71,72,73 
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As noted, it is in the interaction between reader and text that meanings are constructed and 
purposes are achieved. In selecting texts for the PIRLS assessments, the aim is to present a wide range 
of text types within each purpose for reading. The goal is to create a reading experience for students 
participating in each assessment that, as much as possible, is similar to authentic reading experiences 
they may have in and outside of school. 


Reading for Literary Experience 


In literary reading, readers engage with the text to become involved in events, settings, actions, 
consequences, characters, atmosphere, feelings, and ideas, and to enjoy language itself. In order to 
understand and appreciate literature, each reader must bring to the text his or her own experiences, 
feelings, appreciation of language, and knowledge of literary forms. For young readers, literature can 
offer the opportunity to explore situations and feelings they have not yet encountered. 

Events, actions, and consequences depicted in narrative fiction allow readers to experience 
vicariously and reflect upon situations that, although they may be imagined, illuminate those of real 
life. The text may present the perspective of the narrator or a principal character, and a more complex 
text may even have several viewpoints. Information and ideas may be described directly or through 
dialogue and events. Short stories or novels sometimes narrate events chronologically, or sometimes 
make more complex use of time with flashbacks or time shifts. 

The main form of literary texts used in PIRLS is narrative fiction. Given differences in curricula 
and cultures across the participating countries, it is difficult for PIRLS to include some forms of 
literary texts. For example, poetry is difficult to translate and is therefore avoided. 


Reading to Acquire and Use Information 


Informational texts are both read and written for a wide variety of functions. While the primary 
function of informational text is to provide information, writers often address their subject matter 
with different objectives. Many informational texts are straightforward presentations of facts, such as 
biographical details or steps to accomplish a task; however, some informational texts are subjective. 
For example, authors may elect to convey facts and explanations through an expository summary, a 
persuasive essay, or a balanced argument. A reader must bring a critical mind to these texts in order 
to form his or her own opinion. 

In order to best address the various functions of texts, information can be presented differently, 
such as by varying the content, organization, and form. Young students may read informational 
texts that cover a range of content, including those that are scientific, historical, geographical, or 
social. These texts also may vary in the organization of the content conveyed. For example, historical 
facts may be organized chronologically, instructions or procedures sequenced step-by-step, and an 
argument presented logically (e.g., cause and effect, or compare and contrast). 
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Information can be presented in many different formats. Even informational pieces that 
are primarily presented via text may include a table to document facts or a picture to illustrate a 
description. Both print materials (e.g., manuals and newspapers) and websites present a considerable 
amount of information via lists, charts, graphs, and diagrams. In addition, words need not be in the 
form of continuous text, such as in advertisements or announcements, or in sidebars to the text that 
offer supplemental information such as definitions, lists, or timelines. 

Webpages tend to be multimodal in the ways they present information and contain interactive, 
experiential features that are not possible to reproduce in a print format. Multimodal texts utilize 
multiple communicative modes, which are then integrated by the reader in order to extract meaning 
from the text.”4 For example, online text presentations typically integrate the following dynamic 
elements for visual interest or illustration: videos and audio clips; animated graphics; pop-up 
windows with information that only appears by clicking, “hovering” above, or “rolling over” it; and a 
variety of code-based features, such as information that appears and disappears, revolves, or changes 
color. Print-based texts also are frequently multimodal, containing photographs, diagrams, charts, or 
other visual features alongside written text.” 

Looking for and learning from information from the internet involves comprehension of 
information arranged within a complex reading environment. Effective learning when reading online, 
then, necessitates the integration of multiple texts, which may contain contradictory or incomplete 
information.”° Textual elements and attributes, such as source information, relevance to the assigned 
task, and relationships to other sources must be recognized and evaluated in order to integrate texts 
successfully.’778” 

A fundamental component of successful internet research and comprehension is the ability to 
locate information that meets one’s needs. Readers need to be able to find and select the websites 
that will provide the target information, navigate to the relevant web pages, and follow links to 
new websites. Internet searches for information require the additional comprehension demands of 
inferring the potential usefulness of yet unseen texts (e.g., when evaluating search engine results or 
links). In order to begin the search for information, online readers must choose among websites to 
find the one most likely to contain the target information. Once on a given website or page, readers 
must continue to infer the relevance of the various types of information and texts presented, while 
ignoring a barrage of advertising. This may involve self-regulatory processes to maintain focus on the 
task at hand, so as not to be distracted by other interesting topics or advertising. 

The informational texts used in the PIRLS assessments reflect students’ authentic experiences 
with reading informational text in and out of school. Typically, these passages, as well as some of the 
ePIRLS websites, have been written by authors who understand writing for a young audience, and 
are provided by the participating countries as representative of the informational materials their 
students read. 
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Processes of Comprehension 


Different reading situations require readers to construct meaning in different ways. Therefore, PIRLS 
assesses four broad-based processes of comprehension typically used by fourth grade readers: focus 
on and retrieve explicitly stated information; make straightforward inferences; interpret and integrate 
ideas and information; and evaluate and critique content and textual elements. Transcending 

these processes are the metacognitive processes and strategies that allow readers to examine their 
understanding and adjust their approach.®°§! 82838485 In addition, the knowledge and background 
experiences that readers bring to reading equip them with an understanding of language, texts, and 
the world, through which they filter their comprehension of the material.°°878889.90.91 

Construction of meaning in online environments requires a blending of new digital literacies 
with the reading comprehension processes required for traditional offline (i.e., print) reading. ePIRLS 
assesses students’ reading achievement when the conceptualization of the PIRLS passages is expanded 
to include a series of interconnected web pages with many different kinds of visual information, such 
as photos, graphs, charts, and maps, in addition to dynamic features such as videos, animations, and 
pop-up windows. 

In PIRLS and ePIRLS, the four comprehension processes are used as a foundation for developing 
the comprehension questions which are based on each reading passage (or set of passages) or task. 
Across the passages, the variety of questions measuring the range of comprehension processes enables 
students to demonstrate a range of abilities and skills in constructing meaning from written texts. 

In thinking about assessment questions, there is, of course, a substantial interaction between the 
length and complexity of the text and the sophistication of the comprehension processes required by 
the reading task. Initially, it may seem that locating and extracting explicitly stated information would 
be less difficult than, for example, making interpretations across an entire text and integrating those 
interpretations with external ideas and experiences. However, texts and tasks can vary with regard to 
length, syntactic complexity, abstractness of ideas, organizational structure, and cognitive demand. 
Thus, the nature of the text impacts the complexity of the questions asked, across and within the four 
types of comprehension processes. 


Focus on and Retrieve Explicitly Stated Information 


Readers vary the attention they give to explicitly stated information in the text.°»? Some ideas in 
the text may elicit particular focus and others may not. For example, readers may focus on ideas 
that confirm or contradict predictions they have made about the text’s meaning or that relate to 
their general purpose for reading. In addition, readers often need to retrieve information explicitly 
stated in the text to answer a question they bring to the reading task, or to check their developing 
understanding of some aspect of the text’s meaning. 
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As summarized from Kintsch and Kintsch,” retrieval results in a sequence of idea units that 
can be interrelated to form the microstructure of part or all of a text. In addition, there are relations 
among various sections of a text called the macrostructure. The microstructure and macrostructure 
form the textbase, which is very close to the text but an important foundation to developing real 
understanding. The ability to focus on and retrieve explicitly stated information is key to constructing 
the textbase (even though inferences often are necessary for coherence). Typically, this type of text 
processing requires the reader to focus on the text at the word, phrase, and sentence level in order 
to construct meaning.**** Also, constructing the textbase macrostructure may require the reader to 
retrieve pieces of information from several pertinent locations in the text to construct the organizing 
feature of how information is being presented or the summary of a narrative. 

Successful retrieval requires fairly immediate or automatic understanding of the words, phrases, 
or sentences,*” in combination with the recognition that they are relevant to the information sought. 
Interestingly, printed texts are likely to be initially read and processed at micro-level, whereas online 
search strategies may benefit from initial macro-processing before the reader can focus on the 
sentence, phrase, or part of the graphic that has the information.°*”” 

In classifying items, it is essential to examine the item stem and correct response in relation to 
the text. If the item stem and the correct response both use exact words from the text and are located 
with a sentence or two of each other, the item is classified as “Focus and Retrieve.” If some synonyms 
are used, the item still is “Focus and Retrieve.” 

Reading tasks that may exemplify this type of text processing include the following: 


e Identifying and retrieving information that is relevant to the specific goal of reading; 
e Looking for specific ideas; 

e Searching for definitions of words or phrases; 

e Identifying the setting of a story (e.g., time and place); 

e Finding the topic sentence or main idea (when explicitly stated); and 


e Identifying specific information in a graphic (e.g., graph, table, or map). 


Make Straightforward Inferences 


As readers construct meaning from text, they make inferences about ideas or information not 
explicitly stated.!°° Making inferences allows readers to move beyond the surface of texts and to resolve 
the gaps in meaning that often occur in texts. Some of these inferences are straightforward in that they 
are based primarily on information that is contained in one place in the text—readers may merely 
need to connect two or more ideas or pieces of information. The ideas themselves may be explicitly 
stated, but the connection between them is not, and thus must be inferred. Furthermore, despite the 
inference not being explicitly stated in the text, the meaning of the text remains relatively clear. 
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Skilled readers often make these kinds of inferences automatically.!°! They may immediately 
connect two or more pieces of information, recognizing a relationship even though it is not stated 
in the text. In many cases, the author has constructed a text to lead readers to an obvious or 
straightforward inference. For example, the action(s) of a character at a point in the story may clearly 
point to a particular character trait, and most readers would arrive at the same conclusion about that 
character's personality or viewpoint. 

With this type of processing, readers typically focus on more than just word-, phrase-, or 
sentence-level meaning, but the focus is on local meaning residing within one part of the text. As 
noted above, there are some instances especially in online reading, when readers may need to use 
macro-processing and then micro-processing to find information across a website or a text. Using the 
processes together with success often involves making some inferences about the best approaches to 
use in searching for information. 

Online reading requires a considerable amount of inferencing, beginning with identifying those 
websites and webpages most likely to contain the information of interest. Readers also may infer 
whether it is necessary or useful to follow a link to another page. 

When classifying items, if the item stem and correct response use paraphrases of the original 
phrases or sentences in text then the item is classified as “Straightforward Inferencing.” This can 
mean that new vocabulary is introduced in either the stem or multiple-choice responses, but the 
items still are considered inference items. Also, if the correct answers to the item are located in several 
places within the text but the item stem and the correct response both use exact words from the text, 
then the item is classified as inferencing. 

Reading tasks that may exemplify this type of text processing include the following: 


e Inferring that one event caused another event; 
e Giving the reason for a character's action; 
e Describing the relationship between two characters; and 


e Identifying which section of the text or website would help for a particular purpose. 


Interpret and Integrate Ideas and Information 
As with the more straightforward inferences, readers who are engaged in interpreting and integrating 
ideas and information in text may focus on local or global meanings, or may relate details to overall 
themes and ideas. In any case, these readers are making sense of the author’s intent and developing a 
more complete understanding of the entire text. 

As readers interpret and integrate, they are attempting to construct a more specific or more 
complete understanding of the text by integrating personal knowledge and experience with meaning 
that resides within the text.!°? For example, readers may draw on experience to infer a character’s 
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underlying motive or to construct a mental image of the information conveyed. They often need to 
draw on their understanding of the world, as well as their background knowledge and experiences, 
more than they do for straightforward inferences. 

As readers engage in this interpretive process, they are making connections that are not only 
implicit, but that may be open to some interpretation based on their own perspective. Because of this, 
meaning that is constructed through interpreting and integrating ideas and information is likely to 
vary among readers, depending upon the experiences and knowledge they bring to the reading task. 

Using the internet requires the ability to read and digest information from multiple online 
sources.!°> Integrating and synthesizing information across texts is very challenging, even offline, 
because readers need to comprehend not only one text, but consolidate information across two or 
more texts. In the internet environment, this includes information presented via animation and 
videos as well as in pop-up windows and rollover text and graphics. 

Items classified as “Interpret and Integrate Ideas and Information,’ use concepts and 
generalizations not explicitly stated in the text. The new ideas or information may be included in the 
item stem, the acceptable response, or both. A full credit response requires comprehension of the 
entire text, or at least significant portions of it, as well as ideas or information that go beyond the text. 

Reading tasks that may exemplify this type of text processing include the following: 


e Discerning the overall message or theme of a text; 

e Considering an alternative to actions of characters; 

e Comparing and contrasting text information; 

e Inferring a story's mood or tone; 

e Interpreting a real-world application of text information; and 


e Comparing and contrasting information presented within and across texts or websites. 


Evaluate and Critique Content and Textual Elements 


As readers evaluate the content and elements of a text, the focus shifts from constructing meaning to 
critically considering the text itself. Readers engaged in this process step back from a text in order to 
evaluate and critique it. 

The text content, or meaning, may be evaluated and critiqued from a personal perspective or 
with an objective view. This process may require readers to make a justified judgment, drawing on 
their interpretations and weighing their understanding of the text against their understanding of the 
world—rejecting, accepting, or remaining neutral to the text’s representation. For example, readers 
may counter or confirm claims made in the text or make comparisons with ideas and information 
found in other sources. 
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In evaluating and critiquing elements of text structure and language, readers draw upon their 
knowledge of language usage, presentational features, and general or genre-specific features of 
texts.!°4 The text is considered as a way to convey ideas, feelings, and information. 

Readers may reflect on the author's language choices and devices for conveying meaning and 
judge their adequacy. Relying on their understanding of language conventions, readers may find 
weaknesses in how the text was written or recognize the successful use of the author’s craft. Further, 
readers may evaluate the mode used to impart information—both visual and textual features—and 
explain their functions (e.g., text boxes, pictures, or tables). In evaluating the organization of a text, 
readers draw upon their knowledge of text genre and structure. The extent of past reading experience 
and familiarity with the language are essential to each piece of this process. 

For an item to be classified as “Evaluate and Critique,’ an acceptable response to that item 
involves a judgement about some aspect of the text. For example, the item stem can present more than 
one point view where it is possible for students to argue either point of view (or both) based on the 
text or the item stem can ask for a judgement and the evidence to support it. 

Reading tasks that may exemplify this type of text processing include the following: 


e Judging the completeness or clarity of information in the text; 

e Evaluating the likelihood that the events described could really happen; 

e Evaluating how likely an author’s argument would be to change what people think and do; 
e Judging how well the title of the text reflects the main theme; 

e Describing the effect of language features, such as metaphors or tone; 

e Describing the effect of the graphic elements in the text or website; 

e Determining the point of view or bias of the text or website; and 


e Determining an author’s perspective on the central topic. 


The skills required to evaluate and critique online texts are very similar to those required for 
printed text. However, because anyone can publish anything on the internet, readers also must make 
judgments about the credibility of the source of the information as well as determine the perspective, 
point of view, and bias in the text.!°1° In addition, the visual and textual features on the internet 
tend to be much more varied than similar elements of printed text. 

Online reading tasks that may exemplify this type of text processing include the following: 


e Critiquing the ease of finding information on a website; and 


e Judging the credibility of the information on the website. 
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Navigation in ePIRLS 


In its simulated environment, ePIRLS incorporates a set of navigation skills and strategies specifically 
required to locate and use information on the internet. These include the following: 


e Selecting websites that meet a particular information need; and 


e Using online features to locate information within websites (e.g., content tabs, navigation bars, 
graphic icons, and links). 


However, while ePIRLS is designed to simulate an authentic online reading experience, it is 
within a computer-based environment suitable to fourth grade reading levels and a timed assessment. 
In addition, although it is intended to reflect the types of online reading that students are asked to do 
as part of school-based projects, reports, and research assignments, the online environment of the 
ePIRLS assessment is necessarily very limited in comparison to the entire world of the internet. 

While recognizing that being able to locate internet information underlies all of the reading 
processes, the emphasis in ePIRLS is on assessing reading comprehension rather than navigation 
skills. Because students have a range of internet experiences, ePIRLS begins with a brief set of 
directions that covers how to click on tabs and links as well as how to scroll, when necessary. Using 
the device of a teacher avatar, the ePIRLS assessment moves students through the web pages so 
that students have the opportunity to accomplish the reading tasks in the allotted assessment time. 
Also, throughout the assessment, the teacher avatar points students toward particular websites and 
provides additional assistance when students have difficulty locating particular web pages. Although 
the search process is recursive in real life, students that have difficulty finding the correct web pages 
are automatically moved along to the pages by the teacher avatar after a certain amount of time, and 
this information is tracked by the ePIRLS computer-based assessment. 


Selecting PIRLS Passages and ePIRLS Online Texts 


The PIRLS reading passages, as well as the ePIRLS online reading texts, undergo extensive review 
by the Reading Development Group and the National Research Coordinators. Considerable effort is 
expended to ensure that the texts and websites have the following characteristics: 


e Clarity and coherence; 

e Appropriate content across countries and cultures; 

e Interesting, engaging content for a wide range of students; and 

e Adequate basis for assessing the full range of comprehension processes. 


In order to reflect the goal of approximating an authentic reading experience in the assessment, 
the reading passages—whether presented digitally or in printed formats as well as the simulated 
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online materials—must be typical of those read by students in their everyday experiences and reflect 
students’ authentic reading experiences, in and outside of school. In order to help achieve this goal, 
the texts are typically provided and reviewed by the participating countries to be representative of the 
literary and informational materials their students read. 

The time constraints of the assessment situation place some limits on the length of texts, because 
students need time to read the entire passage and answer comprehension questions. Consistent 
with the range in difficulty across PIRLS, the passage length generally averages from about 500 to 
800 words. However, length will vary somewhat because other text characteristics also affect rate of 
reading. 

With the transition to a digital environment, the aim is to increase the diversity of text types 
included in PIRLS 2021. For example, PIRLS could include texts from plays, magazines, and 
newspapers as well as traditional letters, emails, and short messages. Also, information can be 
presented in many different formats. Even informational pieces that are primarily presented via text 
may include a table to document facts or a picture to illustrate a description. Both print materials and 
websites present a considerable amount of information via lists, charts, graphs, and diagrams. Hybrid 
texts are not new, but there have been developments that have proliferated due to rapid changes in 
communication styles and modes brought about by new media and digital texts. 

The ePIRLS online informational reading tasks in science or social studies are adapted from 
internet websites. Each task involves approximately three different websites totaling about five to ten 
web pages. Reflecting the fact that online reading often involves sorting through more information 
than is actually necessary to achieve one’s goal, the texts contained in an ePIRLS assessment task 
average about 1000 words in total. 

Clarity and coherence are essential criteria for PIRLS texts. Typically, the passages and websites 
have been written by successful authors who understand writing for a young audience, such that 
the texts have an appropriate level of linguistic features and density of information. In the context 
of an international study, attaining authenticity in assessment reading experience may be somewhat 
constrained by the need to translate the texts into numerous languages. Thus, care is taken to choose 
texts that can be translated without loss of clarity in meaning, or in potential for student engagement. 

In selecting texts for use in an international reading assessment, it is crucial to pay close 
attention to the potential for cultural bias. Texts that depend heavily on culture-specific knowledge 
are automatically excluded. Text selection thus involves collecting and considering texts from as 
many of the participating countries as possible. The goal is for the texts to be universally applicable 
across cultures, and for the set of texts in the assessment to vary as widely as possible across nations 
and cultures, such that no country or culture is overrepresented in the assessment texts. The final 
selection of texts is based, in part, on the national and cultural representation of the entire set of 
assessment texts. 
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The appropriateness and readability of texts for the PIRLS assessment primarily is determined 
through iterative reviews by educators and curriculum specialists from countries participating in 
the assessments. Taking into account fairness and sensitivity to gender, racial, ethnic, and religious 
considerations, every effort is made to select texts that are topic and theme appropriate for the grade 
level and that elicit the full range of comprehension processes. 

Finally, it is extremely important for the texts to be interesting to the greatest number of 
students. As part of the field test, students routinely are asked how well they like each of the texts, and 
a high level of positive response is fundamental for a text to be selected for the final set of assessment 
instruments. 
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CHAPTER 2 


PIRLS 2021 Context Questionnaire 
Framework 


Ina V.S. Mullis, Michael O. Martin, and Jenny Liu 


Overview 


In addition to measuring trends in students’ achievement in reading comprehension, PIRLS has the 
important role of collecting information about students’ home and school contexts for learning to 
read. Decades of educational research, including four assessment cycles of PIRLS, have found that 
the extent of students’ opportunities to learn and the effectiveness of their learning environments 
substantially impact their reading achievement, with more opportunities and supportive 
environments associated with higher achievement. 

In each PIRLS assessment to date (2001, 2006, 2011, and 2016), the high-quality measure of 
students’ reading achievement together with extensive information about students’ in- and out-of- 
school experiences while learning to read have provided an important resource for research into how 
to improve reading education. More recently, with trend data from the series of PIRLS assessments, 
relating changes in students’ achievement to changes in educational policies or practices can be a 
powerful source of evidence for determining if the approaches are beneficial. 

As with each previous assessment, the goal of PIRLS 2021 is to build on the existing research 
base and make PIRLS data even more useful in helping educational policymakers and practitioners 
raise students’ levels of achievement. This involves assigning a high priority to continuing to collect 
some information to maintain and extend trends, but also means making sure to “keep up with the 
times” and address new areas of research and policy relevance. 

The PIRLS 2021 Context Questionnaire Framework articulates the information to be collected via 
the PIRLS 2021 Questionnaires, providing brief rationales. The framework is organized according to 
five broad influences on students’ reading development: home contexts, school contexts, classroom 
contexts, student attributes, and national contexts. 
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The Data Collection Instruments 


Consistent with prior assessments, PIRLS 2021 collects the data about students’ contexts for 
learning to read through questionnaires administered to various participants in the education 
process, including the students’ parents and caregivers, the principals of the students’ schools, and 
their reading teachers. The students themselves also complete a questionnaire. Taken together, the 
information from these four questionnaires provides the data for learning about students’ home, 
school, and classroom experiences. For national contexts, PIRLS 2021 will update the PIRLS 
encyclopedia that is published with each assessment to describe each country’s approach to reading 
education. As with previous assessments, representatives from each participating country provide 
the contents of the PIRLS 2021 Encyclopedia. They complete a questionnaire about reading education 
policies at the national level and contribute a chapter to the encyclopedia describing the country’s 
reading curriculum and instructional practices. 

The four PIRLS 2021 questionnaires that provide data that can link to each student's reading 
achievement are described below. 


e The Home Questionnaire, entitled the Learning to Read Survey, is addressed to the parents or 
primary caregivers of each student taking part in the PIRLS 2021 data collection. This short 
10-to-15-minute questionnaire solicits information on the home context, such as languages 
spoken in the home, parents’ reading activities and attitudes toward reading, and parents’ 
education and occupation. The questionnaire also collects data on the students’ early childhood 
education, early literacy and numeracy activities, and the child’s reading readiness at the 
beginning of primary school. 


e The Teacher Questionnaire, completed by students’ reading teachers, gathers information 
about classroom contexts for reading instruction, such as characteristics of the class, reading 
instructional time, and instructional approaches. The questionnaire also asks about teacher 
characteristics, such as their career satisfaction, education, and recent professional development 
activities. This questionnaire requires about 35 minutes to complete. 


e The School Questionnaire, completed by the principal of each participating school, asks 
about school characteristics, such as student demographics, the school environment, and the 
availability of school resources and technology. It is designed to take about 30 minutes. 


e The Student Questionnaire, given to all students once they have completed the reading 
assessment, collects information on students’ home environment, such as books in the home and 
other home resources for learning; student experiences in school, including feelings of school 
belonging and whether they are victims of bullying; and students’ reading instruction. Students 
also are asked about their out-of-school reading habits and attitudes toward reading. The 
student questionnaire requires up to 30 minutes to complete. 
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The Development Process 


In characterizing students’ educational contexts for learning to read, PIRLS focuses on policy relevant 
and malleable attributes of the home and school that can be used to help interpret differing levels of 
reading achievement across countries as well as trends from assessment to assessment. 

To summarize questionnaire data more reliably, PIRLS has the practice of developing sets of 
questions related to a topic of interest and combining the results into scales rather than reporting 
the results of the questions one by one. This practice has evolved from the early assessments and 
PIRLS 2011 introduced using item response theory (IRT) methods to construct the scales.1 A number 
of the PIRLS scales measure constructs that are positively related to student achievement, such 
as parents’ attention to early literacy activities and school safety. First in PIRLS 2016, and now in 
PIRLS 2021, work continues on improving the content and measurement properties of the context 
questionnaire scales. 

The TIMSS & PIRLS International Study Center works with the PIRLS Questionnaire 
Development Group (QDG) and National Research Coordinators (NRCs) from the participating 
countries to update the context questionnaire framework and the questionnaires for each successive 
PIRLS assessment. This includes addressing new topics, refining scales, and deleting topics that 
are no longer useful. Development for PIRLS 2021 began in February 2018, when the PIRLS 
2021 NRCs from the participating countries made suggestions for updating the questionnaires. 

Staff at the TIMSS & PIRLS International Study Center drafted an updated context questionnaire 
framework and updated the home, school, teacher, and student questionnaires based on the NRCs’ 
recommendations. The PIRLS 2021 QDG reviewed the updated framework and questionnaires at its 
first meeting in February 2019. Subsequently, the TIMSS & PIRLS International Study Center staff 
revised the framework for online review by the NRCs prior to publication. The questionnaires were 
revised and reviewed by the NRCs prior to field testing at their third meeting in June 2019. After 
the field test, the QDG and the NRCs will review and finalize the questionnaires for the PIRLS 2021 
data collection. 

The remaining sections of the chapter provide the specifications for the PIRLS 2021 context 
questionnaire data collection. 


Home Contexts 


PIRLS has repeatedly shown the importance of home contexts for fostering children’s reading 
achievement. PIRLS 2021 focuses on two major aspects of the home that have been found to be of 
particular importance: the environment for learning and the emphasis on children’s literacy skills. 
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Environment for Learning 


Home Resources for Learning 


In educational research, the factors related to parents’ or caregivers’ socioeconomic status are 
consistently related to students’ achievement.?3+5° Socioeconomic status is often indicated through 
proxy variables such as parental level of education, income, and occupational class. Since the first 
PIRLS cycle in 2001, PIRLS has reported data on socioeconomic status indicators. In 2011, PIRLS 
developed the Home Resources for Learning scale based on parent and student responses to items 
focusing on parental education and occupation as well as home resources (e.g., books in the home, 
children’s books in the home, and home study supports). The Home Resources for Learning scale 
expands the classic conception of socioeconomic status to include home resources that have the 
potential to facilitate student learning. 


Parents Like Reading 


Parents who like reading and read themselves provide important role models for their children. 
Promoting reading in the home can foster children’s motivation to read as well as their reading 
achievement.”*° Parents own reading behaviors and beliefs about reading can shape their child’s 
motivation to read.!° Socialization can be subtle (e.g., young children seeing adults reading or 

using texts in different ways learn to appreciate and use printed material) and this process can have 
long term effects on a student academic performance.'! PIRLS began collecting information on 
parents’ reading habits and enjoyment in 2001, and in 2011, created the Parents Like Reading scale. 
Indicators of parents liking reading include considering reading an important activity and reading for 
enjoyment often. 


Language Spoken in the Home 


It is common for some students to speak one language at home and another at school, especially 
among immigrant families. In addition, some parents prioritize multilingualism and make great 
efforts to ensure their child is exposed to more than one language in the home. Because learning to 
read is dependent on children’s early language experiences, the language or languages spoken at home 
and how they are used are important factors in reading literacy development.’ If students are not 
fluent in the language of instruction, often there is an initial learning gap because students must learn 
the concepts and content of the curricula through a new language.'*4 PIRLS has collected data on 
student language background at home since the first PIRLS cycle. 
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Emphasis on Literacy 


Early Literacy Activities 

Considerable research, including results from PIRLS, has documented the importance of early 
childhood literacy activities in fostering higher student achievement in reading.'>!*!7!69 Examples 
of these activities include reading books, telling stories, playing with alphabet toys, talking with their 
children, helping children write letters or words, and reading aloud signs and labels. Perhaps the 
most common and important early literacy activity involves adults and older children reading aloud 
to their young children.”°?! By being read aloud to, children are exposed to oral language, which 
also is important for literacy acquisition.?>7+”> Beginning in PIRLS 2001, and in each cycle since 
then, PIRLS has asked parents about their child’s early literacy activities. In 2011, when PIRLS and 
TIMSS were assessed together, the Early Literacy Activities Before Beginning Primary School scale was 
supplemented with questions about early numeracy activities. PIRLS 2021 asks parents how often 
they engaged their child in a range of early literacy as well as numeracy activities. 

Preprimary Education 

Much research has detailed the importance of preprimary education (e.g., preschool, kindergarten, 
early childhood education programs) as an early start toward higher academic outcomes.” High- 
quality preprimary education and other early childhood interventions are especially beneficial for 
disadvantaged students because they can play an important role in breaking the generationally 
repetitive cycle of poverty and low achievement.’””* PIRLS always has gathered information from 
parents about their child’s preprimary school attendance and duration of attendance. 

Early Literacy Tasks When Beginning Primary School 

Quality preprimary education and early literacy activities in the home help children develop 
foundational literacy skills that support higher achievement in later grades. PIRLS has shown that 
children whose parents reported that the children could do basic literacy tasks when beginning 
primary school had higher reading achievement in fourth grade. The Could Do Early Literacy Tasks 
When Beginning Primary School scale created in 2011 asks parents to report on how well their child 
could do certain literacy tasks, such as recognizing most of the letters of the alphabet and writing 
some words. 

Parental Expectations for Their Children’s Education 

Parents convey their expectations to their children and provide educational goals for them.??*° This 
includes parents and children talking about the value of education, discussing future educational and 
occupational expectations for the child, and helping children draw links between schoolwork and its 
real-world applications.*!*? PIRLS asks parents about their expectations for their child’s education. 
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School Contexts 


PIRLS 2021 collects data about important school contexts for learning. This includes school resources 
generally and specifically for reading instruction, the school climate for learning, the degree of 
discipline and safety in the school, and the school’s emphasis on reading instruction. 


School Resources 


School Composition of Student Body 


Socioeconomic Background—Since the Coleman report,** there has been sustained interest 
in how the socioeconomic composition of the student body is related to individual student 
achievement.*+35.3° There is evidence that students from disadvantaged backgrounds may have 
higher achievement if they attend schools where the majority of students are from advantaged 
backgrounds. Some have attributed this association to peer effects—observing a strong student 
achievement relationship between classmates.*” The higher achievement for students in 
socioeconomically advantaged schools also may be partially explained by such schools having 
better facilities and instructional materials. Additionally, in some countries, schools with high 
proportions of disadvantaged students have difficulty attracting highly qualified teachers.**49 PIRLS 
routinely collects information from schools about the socioeconomic composition of their student 
body, including the percentages of students from economically disadvantaged homes and from 
economically affluent homes. 


Languages Spoken in the School—Schools where many students speak a language other than the 
primary language of instruction may need to have policies and resources that provide extra support 
for these students. Since 2001, PIRLS has asked principals to indicate the percentage of students 
in their school who have the language of the PIRLS assessment as their native language. In the 
majority of past PIRLS assessments, academic achievement was higher in schools where there was a 
higher percentage of students who speak the language of instruction as their native language. 


Literacy Skills of Entering Student Body—Preprimary education as well as learning experiences 
in the home play important roles in promoting students’ literacy skills in preparation for primary 
school. Students who enter the first grade well-equipped with literacy skills have a stronger 
foundation for formal reading instruction. Since 2001, PIRLS has collected data on student literacy 
skills upon entering primary school. The Schools Where Students Enter the Primary Grades with 
Early Literacy Skills scale was developed in PIRLS 2011 to determine the percentage of students who 
enter schools with literacy skills based on principals’ reports. Skills in the scale include whether 
students can recognize most of the letters of the alphabet, read some words, and write letters of the 
alphabet. Similarly, as described under Home Contexts, parents also are asked to report on their 
child’s early literacy skills through the Could Do Early Literacy Tasks When Beginning Primary 
School scale. 
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Resources for Reading Instruction 


Adequate working conditions and facilities as well as sufficient instructional resources are important 
for maintaining a favorable learning environment in schools.*° Although “adequacy” in terms of 
resources can be relative, the extent and quality of school resources have been shown to be critical for 
quality instruction.*!4?34 Based on principals’ perceptions of school resource shortages that affect 
the school’s capacity to provide instruction, PIRLS results since 2001 have consistently indicated 
that students in well-resourced schools generally have higher achievement than those in schools 
with resource shortages. PIRLS 2001 began collecting data on general school needs such as school 
supplies, adequate school buildings, and instructional space. In later cycles, PIRLS also asked about 
specific shortages affecting reading instruction such as library resources and educational software/ 
applications (apps). Principals’ reports about school resource shortages are summarized by the 
Instruction Affected by Reading Resource Shortages scale. 

School resources also include having computers or tablets available for students for learning 
purposes. PIRLS regularly creates a student-to-computer ratio for schools. In PIRLS 2016, students 
in schools with available computers had higher achievement than students in schools with no 
computers. 


School Library or Media Center 


The variety and richness of the reading material available to students form the core of students’ 
reading experience in school. Schools with a well-resourced library or multi-media center may be 
well-positioned to promote student reading. Research has shown that students use the library when 
there are books that interest them; therefore, ensuring that there is a variety of current reading 
materials that would be of interest to the students at each grade is essential to promoting reading 
achievement.* Since 2001, PIRLS has collected information on school libraries, asking principals 
whether their schools have a library and for the number of books in the library. Because libraries also 
are becoming multi-media centers, students also can seek information on subjects of interest through 
access to ebooks, digital periodicals, and online resources. In 2016, PIRLS began gathering school 
information on the availability of digital learning resources. 

While school libraries are common in most countries, some countries have moved toward 
classroom libraries, as discussed under Classroom Reading Instruction. 


Principals’ Formal Education 


Principals act as leaders in schools by overseeing school staff, students, and the school environment. 
Research has shown that strong principal leadership can foster student achievement by creating an 
atmosphere of collective efficacy through a positive school climate and trust among teachers.*%* 
Recognizing the crucial role school principals play and the importance of highly trained and 
well-prepared principals, PIRLS 2016 began collecting data on country requirements to become 
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a principal as well as principals’ educational background, including their highest level of formal 
education and qualifications in educational leadership. 

Principals’ Years of Experience 

Principals’ years of experience can contribute to their strength of leadership. PIRLS 2016 began 
asking principals how many years they have been a principal. Because rapid turnover can lead to dips 


in student achievement,**” principals also are asked how many years they have been a principal in 
their current school. 


Size and Urbanicity of Schools 


PIRLS provides data about school size and school urbanicity because these characteristics can impact 
student learning. Schools vary in the size of student enrollment and can be located in very different 
geographical areas (e.g., urban, suburban, rural). An advantage of small schools is that they can 
provide a more intimate learning environment, allowing for more adult support for students through 
meaningful staff and student relationships and individualized learning.*° However, smaller schools 
also may have less supportive infrastructure such as libraries, laboratories, and gymnasia. Depending 
on the country, schools in urban areas may have access to more resources (e.g., museums, libraries, 
bookstores) than schools in rural areas. 


School Climate 


Parents’ Perceptions of Child’s School 


Since 2001, PIRLS has asked parents about their perceptions of their child’s school. In 2016, PIRLS 
created the Parents’ Perceptions of Their Child’s School scale. The scale summarizes parents’ or 
caregivers’ level of agreement with statements about school academics, school safety, and schools’ 
efforts to include them in their child’s education. The 2016 results showed that most parents tended 
to be satisfied with the school their child attended, which is consistent with results from other 
educational surveys.°!? 


School Emphasis on Academic Success 


A school atmosphere of high expectations for academic excellence can contribute to school success. 
Research has shown that there is a positive association between a school’s emphasis on academic 
success and academic achievement.*?** Academic emphasis, collective efficacy in promoting 
academic performance, and trust among school staff, parents, and students, are indicators of 
academic optimism in a school, a characteristic that fosters a positive academic environment.*>*°*” 
Since PIRLS 2011, the School Emphasis on Academic Success scale has asked principals and teachers 
to characterize their school in terms of a series of teacher, parent, and student attitudes and actions 
that can contribute to academic success. In 2016, PIRLS expanded the scale to include more items to 
better capture the construct, asking principals and teachers the degree to which students respect their 
classmates who excel academically, and parental expectations for student achievement. 
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Teacher Job Satisfaction 

Fostering teacher job satisfaction through a favorable work environment is important in retaining 
qualified teachers in the classroom.** Collaboration and support, as well as other social factors such 
as a positive school culture and strong principal leadership can be essential for cultivating teacher job 
satisfaction and retaining teachers.*?*! In contrast, emotional exhaustion from work stress has been 
found to be negatively related to teacher job satisfaction.* PIRLS began reporting data on teacher job 
satisfaction in 2006 and created the PIRLS Teacher Job Satisfaction scale in 2016. 

Students’ Sense of School Belonging 

Students’ sense of school belonging, also referred to as school connectedness, has been found to 
contribute to their general well-being.* Students with a strong sense of school belonging feel safe 

at school, enjoy school, and have a good relationship with teachers. The Students’ Sense of School 
Belonging scale was developed in PIRLS 2016, asking students to indicate the extent to which they like 
being in school, how much they feel they belong at the school, and whether teachers are fair to them. 
The 2016 results showed a positive association between school belonging and academic achievement, 
corroborating other research on the subject.®°°” PIRLS 2021 improves the scale by including 
whether students have friends at this school to reflect the importance of positive student relationships 
within the school community for students’ sense of school belonging.** 


School Discipline and Safety 


School Discipline 

Schools with disciplinary issues can experience problems with bullying, classroom disturbance, 
and truancy. The sense of security that comes from having a stable school environment with 

few behavioral problems and little or no concern about student or teacher safety is conducive to 
student learning. Research shows that schools where rules are clear and enforced fairly tend to have 
atmospheres of greater discipline and safety.””7! In past PIRLS assessments, students in schools 
with hardly any discipline problems had higher reading achievement than students in schools with 
disciplinary problems. PIRLS has gathered data on school discipline since 2001, and now includes 
the School Discipline scale developed in 2011. The scale summarizes principals’ reports on school 
discipline based on the degree to which schools have issues with student conduct ranging from 
intimidation or verbal abuse among students to student cheating. 


Safe and Orderly School 


School effectiveness research analyzing PIRLS/TIMSS 2011 data has shown that school safety is an 
important prerequisite for student achievement in many countries.” Respect for individual students 
and teachers, a safe and orderly environment, and constructive interactions among administrators, 
teachers, parents, and students all contribute to a positive school climate and are associated with 
higher student achievement.”*,”4’5 Together the School Discipline scale reported by principals and the 
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Safe and Orderly School scale reported by teachers gather information on safe schools. Teachers are 
asked to indicate the extent to which they agree or disagree with various statements on school safety, 
including whether they feel safe at the school and if they believe school rules are enforced in a fair 
and consistent manner. 
Bullying 
Bullying is repeated aggressive behavior that is intended to intimidate or harm the victim and takes 
a variety of forms, both mental and physical. Bullying causes distress to victims, leads to low self- 
esteem, and makes victims feel like they do not belong.”*’”8 Previous PIRLS reports have shown 
that bullied students tend to have lower reading achievement, aligning with findings of other 
research,’980.81 

PIRLS began collecting data on student bullying in 2001. It developed the student bullying 
scale in 2011, based on students’ reports of how often they experience various bullying behavior 
ranging from being made fun of to being physically hurt. With the prevalence of the internet and 
cyberbullying, the Student Bullying scale was expanded in 2016, and again for 2021. Cyberbullying 
behaviors include being sent nasty or hurtful messages or having nasty or hurtful information shared 
online. Like other forms of bullying, cyberbullying is associated with low self-esteem, distress, and 


poor achievement.®8? 


School Emphasis on Reading Instruction 


Emphasis in Early Grades on Reading Skills and Strategies 


National and school level policies establish the reading curriculum. The extent to which the 
prescribed national reading curriculum is implemented in schools can be reflected in the school 
reading curriculum. In every assessment cycle, PIRLS collects data on schools’ reading curricula 
through the fourth grade by asking principals which grades specific reading skills and strategies are 
first emphasized at their school. These skills range from knowing letters of the alphabet to more 
advanced skills such as determining the author’s perspective or intention. 


Time Spent on Language and Reading Instruction 


The amount of time that teachers have to teach the reading curriculum is a key factor in curriculum 
implementation. PIRLS results show countries vary in the intended instructional time prescribed 

by the curriculum and in the actual time of implementation in the classroom. Research has found 
instructional time to be related to student achievement,* although this relationship may depend on 
how efficiently and effectively instructional time is used.*° Since 2001, PIRLS has collected data from 
teachers on the time spent on reading instruction across the curriculum, and began collecting data 
on language instruction in 2006. Additionally, instructional time at school and the instructional time 
prescribed by the language/reading curriculum are reported by principals and the PIRLS National 
Research Coordinators, respectively. 
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Classroom Contexts 


Because the classroom is the primary locus of instruction, PIRLS 2021 pays particular attention 

to classroom contexts associated with the teaching of reading. These include student engagement, 
teaching strategies for comprehension skills, types of texts assigned, organization for teaching, 
classroom library resources, and classroom assessment. Complementing the introduction of 
digitalPIRLS, PIRLS 2021 collects data on information technology in the classroom, including access 
to digital devices (including desktop computers, laptops, or tablets) for reading instruction, use of 
technology, and instruction in online reading. PIRLS 2021 also collects data on classroom climate and 
on teacher preparation, professional development, and experience. 


Classroom Reading Instruction 


Students Engaged in Reading Instruction 


Student engagement focuses the student’s “in-the-moment” cognitive interaction with the content.** 
Engagement can result from teacher instruction, text discussions with peers, or independent reading. 
The challenge for the teacher is to use effective methods of instruction that maintain/support student 
engagement with the content. Classroom support for engagement involves cognitive activation, a 
clear and well-structured instruction, teacher support through feedback, and teaching adapted to the 
students’ needs and interests. °° Clear instruction is particularly important to ensure students are 
engaged, including explaining the content in an accessible way and gauging student understanding of 
the topic. °°! Additionally, students who find the reading content interesting itself are more likely 
to be engaged during lessons. 

PIRLS developed the Students Engaged in Reading Lessons scale in 2011, which includes items 
about whether the teacher is easy to understand, has clear expectations, gives students interesting 
things to read, supports students’ autonomy, and does a variety of things to help them learn and 
enhance their reading skills. 


Teachers Develop Students’ Reading Comprehension Skills and Strategies 


Reading fluency has been shown to be related to reading comprehension with students having high 
achievement in reading comprehension also displaying high levels of reading fluency.** PIRLS collects 
data on the various ways teachers help students practice fluency, including how often teachers ask 
students to read aloud and silently on their own. 

Students who also develop and utilize various comprehension skills and strategies can have 
a deeper understanding of the text they are reading.” Since the first PIRLS cycle, the teacher 
questionnaire has collected information on the types of reading comprehension skills and strategies 
taught to students including identifying main ideas and making predictions. Additional items 
have been added in subsequent cycles to reflect skills and strategies that are important in reading 
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comprehension, such as determining an author’s perspective or intention and student ability to self- 
monitor their reading. 


Teachers Encourage and Motivate Students to Read 


Fostering student motivation in reading is fundamental for reading teachers, because students who 
are motivated to read more, especially at a young age, become better readers.** Motivation can be 
facilitated, according to self-determination theory,” by creating a supportive environment that fosters 
a sense of relatedness, competence, and autonomy. A classroom environment that is overly controlling 
can stifle student motivation because it removes the student’s sense of autonomy.” One way teachers 
can foster autonomy in reading instruction is by allowing students the opportunity to choose their 
reading material.” Additionally, supportive teacher-student relationships are important in fostering 
student motivation.°®* PIRLS asks teachers about how often they do various activities to encourage 
and motivate students to read such as giving students time to read books of their own choosing and 
encouraging student discussions of the text. 


Types of Texts Assigned 


The reading materials teachers assign to students help shape students’ reading experiences in school. 
With the burgeoning increase in readily accessible information on the internet, there has been a push 
for students to develop the skills to comprehend various forms of informational text. Literary texts 
also serve important purposes by engaging students through personal identification with characters 
in a story and encouraging students to think critically when making predictions or connections in 
the text.°? Because PIRLS assesses student comprehension in both informational and literary texts, 
teachers are asked to report how often they assign various types of informational and literary reading 
materials to their students. 


Organizing Students for Reading Instruction 


Teachers use a variety of ways of grouping students to maximize the effectiveness of their reading 
instruction. Generally, small-group instruction can improve reading ability.!°!°! For example, in 

the guided reading approach to small group reading instruction, teachers form small groups that are 
focused on instruction involving a specific skill or strategy rather than on reading ability in general. 
This type of flexible within-class grouping allows for differentiation in order to address the needs of 
each individual student. Homogeneous grouping by ability is another type of grouping thought to 
support students in learning at a pace that reflects their skills in the subject. However, research has 
found that grouping students according to the same reading ability in elementary school may benefit 
high achieving students but have negative consequences for low performing students.!°*!0> The PIRLS 
teacher questionnaire has routinely collected information on various grouping practices, asking 
teachers the frequency that reading is taught as a whole-class activity, in groups of same- or mixed- 
ability students, or to students individually. 
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Classroom Libraries 


Students who have easy access to reading materials are more likely to read, and for this reason, some 
countries have moved to creating classroom libraries that provide a wide variety of text and text 
types, including digital resources, as well as a special place for independent reading. The presence 
of an organized and readily accessible classroom library encourages students to read! and can 
aid teachers in incorporating literature into instruction and fostering positive reading habits and 
attitudes. However, size and access to classroom libraries can vary depending on the socioeconomic 
composition of the school, with students from disadvantaged backgrounds having access to fewer 
books than students from advantaged backgrounds.! 

In some countries classroom libraries replace school libraries, especially in smaller schools, and 
in others they complement school libraries. PIRLS began collecting information on the level of access 
and size of classroom libraries in 2001. 


Homework 


Homework is one way teachers can extend instruction and evaluate student learning. The amount 
of homework assigned varies both within and across countries, with homework not assigned at all 
to fourth grade students in some countries. Although there are differences across countries, teachers 
who assign homework can discuss the homework in class and provide feedback to students. Since 
2001, PIRLS has asked teachers about how often reading homework is assigned to students and the 
time they expect students to spend on homework. In 2011, PIRLS also began asking teachers how 
homework is used. 


Classroom Assessment 


Teachers have a number of ways to monitor student progress and achievement, including direct 
assessments of what students have learned. PIRLS asks teachers about the types of assessments 
administered to students. Informal assessments such as observing students as they work, asking 
students to answer questions during class, and short written assessments help teachers identify 
needs of particular individuals, evaluate the pace of the presentation, and adapt instruction. Formal 
tests, both teacher-made and standardized, as well as long-term projects, typically are used to make 
important decisions about students’ achievement (e.g., grades). 


Information Technology in the Classroom 


Classroom Access to Computers for Reading Instruction 


Classroom access to computers for reading instruction is becoming increasingly important in 
developing students’ online educational research skills and expanding their literacy competencies 

in online reading. Access to digital resources such as PCs and tablets allow teachers to provide 
instruction in online reading strategies.!°° Since 2001, PIRLS has asked teachers about the availability 
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and access of computers for student use in the classroom. For PIRLS 2021, access includes whether 
the school provides each student with a digital device, whether the class has digital devices that 
students can share, and whether students can bring their own digital devices to class. 


Use of Technology 


Teachers’ attitudes such as self-efficacy toward technology are related to the use of technology in 
the classroom.!°” Teachers who often use technology for instruction are more confident in using 
technology compared to teachers who are less frequent users.!°* PIRLS 2021 gathers information on 
teachers’ level of confidence in using digital devices in instruction. 

Classroom technology can be used to create individualized instruction. While technology use 
in primary grades is often used for remedial purposes, technology also can provide challenging 
enrichment activities for more advanced students.!°° Also new to PIRLS 2021 are data on how often 
teachers use digital devices to support learning for low-performing students, high-performing 
students, students with special needs, and for the whole class. 

Instruction in Online Reading 

Reading instruction includes specific instruction in online reading strategies.!!°!!1!2 Compared 

to print media, reading on the internet poses numerous additional challenges as readers need to 
navigate non-linearly to locate information, evaluate the credibility of online information, and 
synthesize information across websites and modes. Since 2001, PIRLS has collected information 

on how often teachers ask students to perform various computer activities such as read texts on the 
computer or use the computer to write stories. With the development of ePIRLS in 2016, additional 
items were included to gather information on online reading instructional practices and strategies, 
such as whether teachers taught students how to be critical when reading on the internet and 
strategies for reading digital text. 


Classroom Climate 


Classroom Management 


Teachers who are strong classroom managers are able to minimize disruptions to instruction and 
focus instructional time on teaching the curricular content. Research has shown a positive association 
between effective classroom management and student achievement.!!>!!41!5 Measures of classroom 
management examine whether class lessons are disrupted, whether students respect the teacher, and 
whether students behave according to teacher instructions.!!*!!”7 New for PIRLS 2021, information on 
classroom management is gathered through a series of items asking students to indicate the frequency 
with which various threats to effective classroom management such as disruptive noise and student 
interruption occur during reading lessons. 
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Classroom Instruction Limited by Student Attributes 

Student attributes such as hunger or lack of sleep can limit the impact of classroom instruction and 
make some classes more challenging to teach than others. Developed for PIRLS 2016, the Classroom 
Instruction Limited by Student Attributes scale summarizes teachers’ reports about student absences, 
hungry or tired students, and students who do not have the prerequisite foundation of content 
knowledge to learn the reading content. Students also are asked to report whether they feel tired and 
hungry, and how often they are absent from school. 


Teachers’ Preparation 


Teachers’ Formal Education 


Teacher education is critical for effective teaching,'!*!!° with prospective teachers needing coursework 
to gain knowledge in the subjects that they will teach, to understand how students learn, and to 

learn about effective pedagogy for teaching reading. Analysis conducted using PIRLS data showed a 
relationship between teachers’ reading coursework and PIRLS reading achievement.!”° Additionally, 
as reported by PIRLS 2016, many countries have increased their educational requirements for 
primary school teachers and almost all countries now require fourth grade teachers to have a four- 
year degree from a university. In every assessment cycle, PIRLS has collected information on teachers’ 
formal education, areas of study, and national policies and practices on teacher education. Teachers 
are asked about their highest level of schooling completed, main areas of study such as primary/ 
elementary education or secondary education, and the extent to which they studied specific areas 

of reading (e.g., literature, education psychology, reading theory). National Research Coordinators 
report information on national policies and practices on teacher education and the requirements to 
become a teacher. 


Teachers’ Years of Experience 

In addition to teacher education and training, teaching experience is important for teacher 
development, especially in the first years of teaching. !*!2?!23 Research also has found that teachers 
continue to develop pedagogical skills after five years of experience, and that this development can 
positively affect student achievement.!7+!?!26 PIRLS routinely collects data on teachers’ years of 


experience. 
Teachers’ Professional Development 


The PIRLS 2016 Encyclopedia indicates that many countries are increasing efforts to provide teachers 
with professional development opportunities. Professional development can help teachers increase 
their effectiveness and broaden their knowledge and is especially important for exposing teachers 

to recent developments such as curricular changes or new technology for classroom instruction. 

For example, professional development is crucial for training teachers on how to incorporate online 
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reading into their classroom practices.!”” Additionally, for professional development to be effective, 
professional development opportunities should engage teachers through concrete tasks, be sustained 
and ongoing, and provide teachers the space to reflect on their teaching.!”8!”° Support systems also 
should be in place to encourage teachers to partake in professional development.'*° Since 2001, PIRLS 
has asked teachers about their professional development. For PIRLS 2021, the teacher questionnaire 
builds on the professional development topic area and collects information on teacher participation 
and demand in specific areas of professional development (e.g., teaching reading comprehension 
skills and strategies, instruction related to digital literacies). PIRLS 2021 also gathers data on the 

type of professional development (e.g., workshops, seminars, mentor access) that is most useful to 
teachers, and on barriers to teacher participation in professional development activities. 


Student Attributes 


PIRLS 2021 collects data on important student attributes including student reading attitudes and 
student demographics. Students who are motivated to read and have a strong reading self-concept 
tend to have better reading comprehension,"! and cultivating these attitudes also may support 
students in becoming lifelong readers. Results from PIRLS have shown relationships between positive 
reading attitudes and reading achievement. It is noted that achievement and attitudes can be mutually 
reinforcing of one another. Better readers may enjoy and value reading more than poorer readers, 
thus reading more and further improving their skills. Because student demographics are important 
for analyzing reading achievement, PIRLS also collects data on student gender and age. 


Student Reading Attitudes 
Students Like Reading 


Student readers who are intrinsically motivated find reading interesting and enjoyable for its own 
sake. Intrinsic motivation is the “energizer of behavior,’!*? and research has shown that intrinsic 
motivation is more closely related to reading achievement than extrinsic motivations such as praise 
and money.!*3!54 PIRLS has recognized the importance of collecting data on students’ attitudes 
toward reading since its inception. In 2011, PIRLS developed the Students Like Reading scale to 
measure students’ intrinsic motivation to read. Students are asked how much they agree with 
statements about liking more time to read, reading for fun, and learning a lot from reading. The scale 
was revised in 2016 and additional items were included to better measure the construct. 


Students Confident in Reading 


Students tend to have distinct views of their own reading ability, and their self-appraisal is often based 
on their past experiences and how they see themselves compared with their peers.!*5 Students who 
are confident in their ability persevere in completing a school task because they believe they can be 
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successful.!°° PIRLS has asked about students’ self-concept as readers from the beginning. Since 2011, 
reading self-concept has been measured using the Students Confident in Reading scale. Students are 
asked to indicate the extent to which they agree with statements such as “reading is easy” and “T 
usually do well in reading,” 

Familiarity in Using Digital Devices 

According to the PIRLS 2016 Encyclopedia, PIRLS countries are working toward integrating 
technology into instruction across the curriculum to help students develop Information and 
Communications Technology (ICT) and digital literacy skills. With the introduction of ePIRLS in 
2016, PIRLS began asking students about using digital devices. Students were asked how well they 
could use the computer, type, and find information online. Results from 2016 showed a positive 
association between students’ computer use and their achievement in online informational reading, 
which aligned with other similar research.!3”138 PIRLS 2021 includes a new scale measuring students’ 
familiarity with using computers or tablets as well as their confidence in finding information on 

the internet.1°° 

Students Liking of Assessment Passages 

Students who enjoy the reading content are more interested and engaged with the text than students 
who do not enjoy the reading content. PIRLS aims to develop a variety of texts that are interesting to 
a wide range of students. In PIRLS 2016, most students (over 80%) participating in PIRLS liked the 
PIRLS passages and ePIRLS tasks. 


Student Demographics 


Gender 

Over the last four cycles of PIRLS, the gender gap in reading achievement has favored girls over boys 
in the majority of participating countries. In PIRLS 2016, girls outperformed boys in 48 out of the 
50 countries, reflecting a pattern seen in research.'° Similar results favoring girls were found in the 
ePIRLS assessment of online informational reading. 

Age 

Students at different ages may perform differently on PIRLS depending on their academic history. In 
countries where students are admitted to primary school strictly on the basis of age, older students 
may be more skilled in reading comprehension compared to younger peers because of greater 
maturation. However, depending on retention policies, older students who have been held back may 
struggle more with reading comprehension than students who have not been held back. 
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National Contexts 


In every country, the educational system is embedded in a unique configuration of historical, 
economic, and language factors that combine to determine priorities in how the system is organized 
for teaching and learning. Countries participating in PIRLS 2021 contribute information on many 
of these factors through their chapter in the PIRLS 2021 Encyclopedia and by way of the curriculum 
questionnaire. More specifically, countries provide information about the organization of the 
education system and the reading curriculum. 


Organization of Education System 


Country Language(s) of Instruction 


The historical background of language and literacy in a country can influence the challenges 

and instructional practices in teaching students to read. For example, some countries have one 
commonly spoken language, while others are historically multilingual. Immigration also can increase 
language diversity. Multilingual countries across the world have different policies for educating their 
population and specifically for language literacy. Thus, decisions about the language(s) of instruction 
and how to implement those decisions can be very complicated. 


System for Preprimary Education 


Even before children begin formal primary school, they receive considerable exposure to literacy as 
part of their preprimary educational experience (e.g., preschool, kindergarten). Preprimary education 
is an area of investment for many countries. Research findings indicate that attendance in preprimary 
programs can have a positive effect on academic outcomes.'4! The PIRLS curriculum questionnaire 
gathers information on countries’ provisions of early childhood education and preprimary education 
detailing the degree of universal coverage. In PIRLS 2016, almost all participating countries provided 
universal preprimary education for children age 3 or older, and a number of countries also sponsored 
universal programs for children younger than 3 years old. Additionally, the European Union recently 
legislated that member countries should provide universal access to preprimary education!” and 
Norway recently began universal access to early childhood education starting at age 1.1% 

It is also becoming increasingly clear that the effect of preprimary education on later academic 
and life outcomes is dependent on the quality of the preprimary program.'“*14>!“6 PIRLS gathers 
data on any associated curriculum for early childhood and preprimary education. As described 
in the PIRLS 2016 Encyclopedia, preprimary education programs often have curricula that focus 
on children’s physical and socioemotional development and incorporate literacy and numeracy 
pedagogy as well as experiential science activities. 
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Age of Entry and Retention 

Policies about the age of entry into formal education (first year of primary school, ISCED Level 1) 
are important for understanding achievement differences as well as the variation in students’ ages 

across countries at the fourth grade.'4” Students who enter school at an older age are more mature 

at school entry and may be able to cope more easily with complex reading materials from the first 

grade onward. Data on country’s age of entry to primary school are reported by National Research 
Coordinators. 

Additionally, because PIRLS is a grade-based study, PIRLS also gathers information on 
countries’ student promotion and retention policies, an important factor to consider when evaluating 
achievement results. Research has shown that grade retention does not have a positive relationship 
with student achievement or the emotional well-being of the student and is overall inefficient.!4°!” 
For these reasons, many PIRLS countries practice automatic promotion, especially in the primary 
grades.!*° 
Number of Years of School 
More years of required schooling allow more time for students to learn and become more educated 
individuals with higher educational attainment. PIRLS collects data on nationally mandated 


compulsory years of education. In PIRLS 2016, most countries reported beginning compulsory 
education around age 6 and ending compulsory education around age 16. 


Reading Curriculum 


Reading Curriculum in the Primary Grades 


Whether formulated at the national, community, or school level, countries have curricula that define 
what students should be taught, providing expectations for students in terms of the knowledge, skills, 
and attitudes to be developed or acquired through their formal reading instruction. 

Curricular aspects and governing policies particularly relevant to the acquisition of reading 
literacy include the standards or benchmarks established for reading development. A coherent 
progression of instruction and comprehension strategies for reading development includes a change 
in emphasis from decoding to comprehension strategies, and access to a wide variety of reading 
materials, as well as sensible differentiation policies for accelerated readers and struggling readers. 

Especially relevant to the ePIRLS assessment of online informational reading is the extent 
to which a country’s curriculum emphasizes online reading and new media. As documented in 
the PIRLS 2016 Encyclopedia, some countries have transformed their standards, curriculum, and 
assessments in order to address teaching elementary school students digital reading strategies such as 
how to navigate and retrieve online information.!>!!> 

In every assessment cycle, PIRLS National Research Coordinators report on their 
country’s reading curriculum through the PIRLS encyclopedia. Data gathered about the reading 
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curriculum from the curriculum questionnaire has evolved over the cycles to also include how the 
implementation of the curriculum is evaluated, the use of technology in reading instruction, and the 
degree of emphasis placed on digital reading in the curriculum. 


Students with Reading Difficulties 


Countries have taken initiatives to ensure that education is inclusive of all students, providing equal 
educational opportunities to all students, including those with reading difficulties.!* In order for 
appropriate accommodations to be made to student learning and assessment, it is important to 
identify students who need these accommodations. Identification of reading difficulties such as 
dyslexia early in students’ education can help educators determine the best course of action for 
these students and prevent students from falling further behind in their reading development.!** 
PIRLS 2021 gathers information on national provisions regarding diagnostic testing for reading 
difficulties and instruction for children with reading difficulties. 
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CHAPTER 3 


PIRLS 2021 Assessment Design 


Michael O. Martin, Matthias von Davier, Pierre Foy and Ina V. S. Mullis 


Overview 


PIRLS 2021 is a comprehensive assessment of fourth grade students’ reading literacy achievement. 
Conducted on a regular five-year cycle, with each assessment linked to those that preceded it, PIRLS 
provides regular data on trends in students’ reading literacy on a common achievement scale. PIRLS 
2021 begins the transition from paper-and-pencil to digital format, with about half the countries 
choosing to administer the digital format (digitalPIRLS) and half the paper format (paperPIRLS). To 
ensure comparability across formats, digitalPIRLS and paperPIRLS have the same content in terms 
of reading passages and questions, although digitalPIRLS takes advantage of some features and item 
types not available in paper and pencil mode. Countries choosing digitalPIRLS will also administer 
ePIRLS, an assessment of online reading to acquire and use information first conducted as part 
of PIRLS 2016. Similar to previous PIRLS assessments, PIRLS 2021 includes a series of contextual 
questionnaires to gather information about community, home, and school contexts for developing 
reading literacy. 

Based on PIRLS experience in earlier assessment cycles, it is clear that achievement levels 
in reading comprehension vary widely both across and within countries, posing a challenge in 
matching the difficulty of the assessment to the reading ability of the students in every country. 
When an assessment is much too difficult or too easy for a population, little information is available 
to accurately measure performance. To address this challenge, PIRLS in the past has offered less 
difficult versions of its assessment materials that countries could choose to administer, beginning 
with prePIRLS in 2011 and following up with PIRLS Literacy in 2016. These efforts were successful 
in expanding PIRLS coverage of students at the lower end of the ability distribution but required 
separate (although linked) versions of PIRLS and did not address the need for more challenging 
material for higher achieving students. 

PIRLS 2021 addresses the need for a broader range of assessment difficulty and better targeting 
of student ability by adopting a single unified assessment based on a new group adaptive assessment 


design (Click here for the rationale underlying adaptive assessment designs). The new design is based 
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on having three levels of passage difficulty—difficult, medium, and easy—that are combined into 

two levels of booklet difficulty. More difficult booklets are composed of two difficult passages or 

one medium and one difficult passage while less difficult booklets consist of an easy and a medium 
passage or two easy passages. Each country administers the entire assessment, but the balance of 
more difficult and less difficult booklets varies with the reading achievement level of the students in 
the country. For example, a country with higher average reading achievement (average score of 550 or 
above) could assign the more difficult booklets to 70% of its students and the less difficult booklets to 
30% of its students, whereas a country with lower achievement levels (average score of 450 or below) 
could assign the more difficult booklets to 30% of its students and the less difficult booklets to 70%. 

The group adaptive design improves the match between assessment difficulty and student ability 
in each country’s population by having a greater proportion of more difficult booklets in countries 
with relatively high achievement and a greater proportion of less difficult booklets in countries with 
relatively low achievement. Accordingly, the new design maximizes the information obtained from 
the assessment while minimally changing existing procedures and time requirements. 

Although the group adaptive design was developed to provide a better match between 
assessment difficulty and student ability at the country level, it also is possible to apply the approach 
within a country, provided the country has clearly defined subpopulations that differ substantially in 
student achievement. 


Student Population Assessed 


PIRLS assesses the reading literacy of children in their fourth year of formal schooling. This 

student population was chosen for PIRLS because it is an important transition point in children’s 
development as readers. Typically, at this point, students have mastered the basics of learning to read 
and are now reading to learn. In many countries, this also is when students begin to have separate 
classes for different subjects, such as mathematics and science. 

PIRLS defines the fourth year of formal schooling according to the International Standard 
Classification of Education (ISCED) developed by the UNESCO Institute for Statistics. The ISCED 
classification provides an international standard for describing levels of schooling across countries, 
and covers the full range of schooling, from early childhood education (Level 0) to doctoral study 
(Level 8). ISCED Level 1 corresponds to primary education, or the first stage of basic education, and 
is considered to be the first stage of formal schooling. 

The target population for PIRLS is defined as follows: 


The PIRLS target grade should be the grade that represents four years of schooling, 
counting from the first year of ISCED Level 1. 
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The PIRLS target grade is four years after the beginning of Level 1, which is the fourth grade 
in most countries. However, given the linguistic and cognitive demands of reading, PIRLS wants to 
avoid assessing very young children. Thus, if the average age of fourth grade students at the time of 
testing would be less than 9.5 years, PIRLS recommends that countries assess the next higher grade 
(i.e., fifth grade). 


Reporting Reading Achievement 


The PIRLS assessment is designed to provide a complete picture of the reading literacy achievement 
of the participating students in each country. This includes achievement by reading purpose 

and comprehension process as well as overall reading achievement. Consistent with the goal of 

a comprehensive view of reading comprehension, the entire PIRLS 2021 assessment, digital and 
paper versions, consists of 18 reading passages and accompanying questions (known as items), half 
assessing reading for literary experience and half assessing reading to acquire and use information. 
In accordance with the group adaptive design, one third of the passages are relatively difficult, one 
third of medium difficulty, and one third relatively easy. Countries administering digitalPIRLS also 
administer ePIRLS, which consists of five tasks that assess online informational reading. 

In order to keep the assessment burden on any one student to a minimum, each student is 
presented with just two passages, one literary and one informational, according to a systematic 
booklet assembly and rotation procedure, as described in the next section. In digitalPIRLS countries, 
some students also are presented with ePIRLS material, either two ePIRLS tasks, or one digitalPIRLS 
informational passage followed by one ePIRLS task. The PIRLS administration consists of two 
40-minute sessions, one for each passage or task, separated by a short break, and followed by a 
30-minute session for the student questionnaire. Following data collection, student responses to the 
assessment passages are placed on the PIRLS reading achievement scales using item response theory 
methods that provide an overall picture of the assessment results for each country.’ 

PIRLS was designed from the outset to measure trends over time in student reading 
achievement. Accordingly, the PIRLS reading achievement scale provides a common metric on which 
countries can compare their fourth grade students’ progress in reading over time from assessment 
to assessment. The PIRLS achievement scale was established in 2001 so that 100 points on the scale 
corresponded to one standard deviation across all of the countries that participated in 2001, and 
the scale centerpoint of 500 corresponded to the international average across those countries. Using 
passages that were administered in both the 2001 and 2006 assessments as a basis for linking the 
two sets of assessment results, the PIRLS 2006 data also were placed on this scale so that countries 
could gauge changes in students’ reading achievement since 2001. Following a similar procedure, 
the PIRLS 2011 and PIRLS 2016 data also were placed on the PIRLS scale, as will be the data 
from PIRLS 2021. This will enable countries that have participated in PIRLS since its inception to 
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have comparable achievement data from 2001, 2006, 2011, 2016, and 2021, and to plot changes in 
performance over this 20-year period. 

The PIRLS reading achievement scale is an overall measure of reading proficiency that includes 
both reading purposes and processes of comprehension. However, in addition to the overall scale, 
PIRLS also provides separate achievement scales on the same metric for purposes for reading and for 
processes of comprehension. More specifically, there are two scales for reading purposes: 


e Reading for literary experience; and 
e Reading to acquire and use information. 
In addition to these, there also are two scales for processes of reading comprehension: 
e Retrieval and straightforward inferencing; and 
e Interpreting, integrating, and evaluating.” 


Countries participating in digitalPIRLS also administer ePIRLS; so, in addition to the usual 
PIRLS overall reading achievement results and results by reading purpose and comprehension 
process, in countries participating in digitalPIRLS, student achievement will also be reported for 
online informational reading. The ePIRLS online reading achievement scale was established in 
2016 to enable countries to examine their students’ online reading performance relative to their 
performance on the PIRLS reading achievement scales. 


PIRLS 2021 Group Adaptive Design 


Implementing the group adaptive design in PIRLS 2021 required grouping the assessment passages 
and items into three levels of difficulty—easy, medium, and difficult—with three literary and three 
informational passages at each level of difficulty. Of the 18 passages needed for the design, 12 were 
administered previously in PIRLS 2016 and were available to support the measurement of trends and 
six were developed and field tested for first time use in PIRLS 2021. 

In 2016, the main PIRLS assessment was accompanied by PIRLS Literacy, which was a reading 
assessment similar to PIRLS in size and scope but less difficult, with shorter and less demanding 
passages and easier items. Taking PIRLS and PIRLS Literacy together, there were eight passages (four 
literary and four informational) that appeared only in PIRLS, four shared passages (two for each 
purpose) that appeared in both PIRLS and PIRLS Literacy, and eight passages (again four for each 
purpose) that appeared in PIRLS Literacy only. The 12 trend passages for 2021 were chosen from 
among these passages, with the difficult passages drawn from the PIRLS only passages, the medium 
passages from the passages shared between PIRLS and PIRLS Literacy (with one exception), and the 
easy passages from those used in PIRLS Literacy only. 


* Retrieval and straightforward inferencing combines items from the Focus on and Retrieve Explicitly Stated Information and Make Straightforward Inferences 
comprehension processes. Similarly, interpreting, integrating, and evaluating is based on items from the Interpret and Integrate Ideas and Information and 
Examine and Critique Content and Textual Elements processes. 
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Exhibit 1: Reading purpose and difficulty level for 18 PIRLS 2021 passages 
Passages 
LitD1 (06) Shiny Straw (06) 
Difficult LitD2 (16) Oliver and the Griffin (16) 
LitD3 (21) New LitD3 Passage (21) 
LitM1 (16) Pemba Sherpa (16) 
Literary Medium LitM2 (21) New LitM2 Passage (21) 
LitM3 (11) The Empty Pot (11) 
LitE1 (21) New LitE1 Passage (21) 
Easy LitE2 (11) The Summer My Father Was 10 (11) 
LitE3 (16) Library Mouse (16) 
InfD1 (11) Where's the Honey? (11) 
Difficult InfD2 (16) Icelandic Horses (16) 

IndD3 (21) New InfD3 Passage (21) 
InfM1 (16) How Did We Learn to Fly? (16) 

Informational Medium InfM2 (21) New InfM2 Passage (21) 
InfM3 (06) Sharks (06) 
InfE1 (21) New InfE1 Passage (21) 

Easy InfE2 (11) Training a Deaf Polar Bear (11) 

InfE3 (16) Hungry Plant (16) 


* The number in parentheses is the assessment year in which the passage was first used. 


Exhibit 1 shows how the existing trend passages fit into the purpose-by-difficulty level scheme, 
and also where the new passages and items belong. Of the six new passages, three will be literary and 
three informational, with one of each in each of the difficulty categories. The exhibit also includes a 
passage label for each passage to facilitate assignment of passages to booklets. 


Passage Difficulty Level 


For the design to be effective, it is necessary that there be distinct differences between the average 
difficulties of the passages in each of the passage groups. For example, reasonable difficulty goals in 
terms of average percent correct across the student population would be 40% for the difficult group, 
60% for the medium group, and 80% for the easy group. New passages developed for PIRLS 2021 will 
aim for these difficulty levels, but there is less flexibility with the existing, trend passages, which make 
up two-thirds of the passage total. 
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2021 
Exhibit 2: Average difficulties of existing trend passages from 2016 and target difficulties for 2021 (average 
percent correct) 
Difficulty of Trend Passages sg: 

Passage Level emaols Target Difficulty for 2021 
Difficult 55% 50% 

Medium 68% 65% 

Easy 80% 80% 


As shown in Exhibit 2, the difficulties of the existing medium and difficult trend passages (68% 
and 55%, respectively) are higher than the long-term goals (60% and 40%, respectively), especially 
for the difficult passages. However, by combining the existing passages with new passages developed 
to be closer to the target difficulty, it will be possible to make progress in 2021 toward these long- 
term goals. Exhibit 2 shows the interim target difficulties for 2021, which should be attainable given 
the mix of existing and new passages. Further progress will be made toward the long-term goals in 
PIRLS 2026, as more of the trend passages are replaced by new, more targeted passages. 


Booklet Design 


In the PIRLS assessment, each student is randomly assigned a test booklet (or booklet equivalent in 
digitalPIRLS) consisting of two passages and their items. In PIRLS 2021, the 18 passages are arranged 
into 18 booklets of two passages each, with each passage appearing in two booklets and paired with 

a different passage each time. Exhibit 3 summarizes the passage pairs that make up each booklet. 

The direction of the arrows shows which passage comes first in the booklet. For example, an arrow 
points from passage InfM1 to LitD1, indicating that these two passages share a booklet, with InfM1 
preceding LitD1. Note that when passages of different difficulties are paired in the same booklet, the 
easier of the two always comes first. 


Exhibit 3: Passage pairings for each assessment booklet 


Passages and Links 


Reading Purpose Difficult Medium Easy 
Passages & Items Passages & Items Passages & Items 


LitD1 (11) LitM1 (16) LitE1 ( 
FT TIE 
[oss SCS Cig SOOT 8 = 
(R01 i SSCS ttt 7 SSCS tes 077 
See re ee, 


InfD3 (21) InfM3 (11) InfE3 (16) 


Literary 


Informational 
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The 18 booklets are divided into two levels of difficulty, as follows: 


e More difficult booklets (9) composed of either two difficult passages or one medium and one 
difficult passage 


e Less difficult booklets (9) composed of two easy passages or one easy and one medium passages. 
Exhibit 4 shows the passage assignments for the 18 booklets, with booklets 1-9 being the more 
difficult booklets and booklets 10-18 the less difficult ones. 


Exhibit 4: Assessment booklets with passage assignments 


Student Assessment 
12YoYo) 4 (3 4 


Booklet 1 InfM1 (16) LitD1 (11) 
Booklet 2 LitD3 (21) InfD2 (16) 
Booklet 3 LitM1 (16) InfD1 (11) 
More Booklet 4 InfM2 (21) LitD2 (16) 
Difficult Booklet 5 LitD1 (11) InfD3 (21) 
Booklets Booklet 6 LitM2 (21) InfD2 (16) 
Booklet 7 InfM3 (11) LitD3 (21) 
Booklet 8 InfD1 (11) LitD2 (16) 
Booklet 9 LitM3 (11) InfD3 (21) 
Booklet 10 LitE1 (21) InfM1 (16) 
Booklet 11 InfE2 (11) LitM2 (21) 
Booklet 12 InfE1 (21) LitE3 (16) 
inves Booklet 13 LitE2 (11) InfM2 (21) 
Difficult Booklet 14 InfE3 (16) LitM3 (11) 
Booklets Booklet 15 LitE1 (21) InfE2 (11) 
Booklet 16 LitE3 (16) InfM3 (11) 
Booklet 17 InfE1 (21) LitM1 (16) 
Booklet 18 LitE2 (11) InfE3 (16) 


Exhibit 5 also presents the passage assignments for each booklet, this time showing where the 12 
existing trend passages belong and where the new passages developed for 2021 will go. 
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Exhibit 5: Assessment Booklets with Trend and New Passage Assignments 


Student Assessment 
Booklets 


Booklet 1 How Did We Learn to Fly? (16) Shiny Straw (06) 
Booklet 2 New LitD3 Passage (21) Icelandic Horses (16) 
Booklet 3 Pemba Sherpa (16) Where's the Honey? (11) 
More Booklet 4 New InfM2 Passage (21) Oliver and the Griffin (16) 
Difficult Booklet 5 Shiny Straw (06) New InfD3 Passage (21) 
Booklets Booklet 6 New LitM2 Passage (21) Icelandic Horses (16) 
Booklet 7 Sharks (06) New LitD3 Passage (21) 
Booklet 8 Where's the Honey? (11) Oliver and the Griffin (16) 
Booklet 9 The Empty Pot (11) New InfD3 Passage (21) 
Booklet 10 New LitE1 Passage (21) How Did We Learn to Fly? (16) 
Booklet 11 Training a Deaf Polar Bear (11) New LitM2 Passage (21) 
Booklet 12 New InfE1 Passage (21) Library Mouse (16) 
ieee Booklet 13 The Summer My Father Was 10 (11) New InfM2 Passage (21) 
Difficult Booklet 14 Hungry Plant (16) The Empty Pot (11) 
Booklets Booklet 15 New LitE1 Passage (21) Training a Deaf Polar Bear (11) 
Booklet 16 Library Mouse (16) Sharks (06) 
Booklet 17 New InfE1 Passage (21) Pemba Sherpa (16) 
Booklet 18 The Summer My Father Was 10 (11) Hungry Plant (16) 


Booklet Assignment within Countries 


To ensure that the same assessment is conducted in every country, all 18 passages are distributed 


in every country, but with varying proportions of the more and less difficult booklets depending 


on the average reading ability of the student population. This is estimated based on performance 


in prior PIRLS assessments, or in the field test for countries participating for the first time. Higher 


performing countries sample proportionally more of the more difficult booklets while lower 


performing countries sample proportionally more of the less difficult booklets, with the goal of a 


better match between assessment difficulty and student ability in each country. 


Exhibit 6 illustrates the differential booklet assignment plan for higher, middle, and lower 


performing countries. Countries with higher average performance, above 550 on the PIRLS 


achievement scale, would randomly assign proportionally more of the more difficult booklets, e.g., 


70%, and fewer of the less difficult booklets, e.g., 30%. Countries with performance between 450 and 


550 would assign equal proportions of more and less difficult booklets, and countries with lower 


average performance, below 450 on the PIRLS scale, would assign proportionally fewer of the more 
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difficult booklets (30%) and more of the less difficult booklets (70%). 
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Exhibit 6: Booklet assignment plan for higher, middle, and lower performing countries 


70” More 30” Less 
Difficult Difficult 


50” More 50” Less 
Difficult Difficult 


30” More 70” Less 
Difficult Difficult 


Integrating ePIRLS with digitalPIRLS 


The ePIRLS assessment of online informational reading in 2021 consists of five tasks presented 

by computer or tablet. With the guidance of a teacher avatar, students navigate within and across 
webpages to answer questions, explain relationships, and interpret and integrate information. Three 
of the tasks were administered as part of ePIRLS 2016 and are available for measuring trends and two 
were developed for PIRLS 2021. As a step toward the future further integration of PIRLS and ePIRLS, 
in digitalPIRLS countries ePIRLS tasks are included in the booklet assignment rotation scheme, as 
are a number of “hybrid” booklets consisting on one digitalPIRLS informational passage followed 

by one ePIRLS task. This means that students participating in digitalPIRLS may be presented with 
one of three booklet types: a regular booklet with two digitalPIRLS passages, an ePIRLS booklet with 
two ePIRLS tasks, or a hybrid booklet with one digitalPIRLS informational passage followed by one 
ePIRLS task. 


digitalPIRLS Booklet Assignment Rotation Schemes 


The basic assignment rotation scheme for digitalPIRLS is designed to match that of paperPIRLS, 
and so has the same 18 passages arranged in the same 18 booklets as paperPIRLS. In paperPIRLS 
these 18 booklets are distributed among the students in sampled classes using a systematic random 
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assignment process that ensures that the proportions of more and less difficult booklets conform to 
the rates established for the country. digitalPIRLS follows a similar assignment process for the 18 
regular booklets, but also includes provision for the ePIRLS and hybrid booklets. 

Similar to the ePIRLS assignment rotation scheme used in 2016, each of the five ePIRLS tasks for 
2021 is paired with each of the others, with each task appearing in both the first and second positions 
in each booklet. This results in 20 distinct ePIRLS booklets. In the hybrid booklets, each of the nine 
digitalPIRLS informational passages is paired with each of the five ePIRLS tasks, resulting in 45 
hybrid booklets. In each one, the digitalPIRLS passage precedes the ePIRLS task. 

Combining the 18 digitalPIRLS booklets, the 20 ePIRLS booklets, and the 45 hybrid booklets 
results in a total of 83 booklets to be rotated among the sampled students. However, including each 
booklet type in the rotation at the same rate would result in too few digitalPIRLS booklets relative 
to the other booklet types, and especially relative to the number of such booklets in paperPIRLS 
countries. To avoid this imbalance, digitalPIRLS uses a 27 booklet rotation that includes all 18 
digitalPIRLS booklets, six ePIRLS booklets, and three hybrid booklets. The assignment of booklets 
to individual students is conducted by IEA’s WinW3S within school sampling software to ensure 
accurate implementation of the rotation scheme. 

Beginning with a booklet chosen at random by WinW33S, the first iteration of this rotation 
involves all 18 digitalPIRLS booklets, booklets 1 to 6 of the 20 ePIRLS booklets, and booklets 1 to 3 of 
the hybrid booklets. The second iteration includes the 18 digitalPIRLS booklets again, together with 
ePIRLS booklets 7 to 12 and hybrid booklets 4 to 6. This pattern continues with the next iteration 
including the 18 digitalPIRLS booklets once again, together with ePIRLS booklets 13 to 18 and hybrid 
booklets 7 to 9, and so on throughout the country’s entire student sample. This booklet assignment 
scheme results in two thirds of the student sample responding to digitalPIRLS booklets, two ninths to 
ePIRLS booklets, and one ninth to hybrid booklets. 

The PIRLS 2021 group adaptive design represents a change from the design of previous PIRLS 
assessments, where booklets of approximately equal difficulty were distributed at random among 
students in sampled classes with equal probability in each country. By allowing booklets that vary 
in difficulty to be assigned at country specific rates, the group adaptive design aims to improve the 
accuracy of measurement in countries participating in PIRLS. 
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APPENDIX A 


Rationale for Group Adaptive Designs in 
International Large Scale Assessment 


Different test forms (booklets) are commonly used in large scale international assessments such as 
PIRLS to balance respondent burden and content coverage. Country level group adaptive assessment 
designs extend this approach through targeted sampling of booklets to provide better coverage 

of the diverse range of ability distributions encountered in such assessments. This can increase 
student motivation and reduce item level nonresponse. The PIRLS 2021 approach is designed to 
minimally change existing procedures and time requirements, while using prior data about country 
performance to maximize the information obtained from the assessment. 

The basic idea behind adaptive assessment is that in order to enable any type of measurement, 
tasks must not be too easy or too difficult for the target population. If the tasks given to a sample of 
test takers are too difficult, nobody (or almost nobody) will be able to solve them. Similarly, if the 
tasks are too easy, everybody will answer them all correctly. In each of these situations all test takers 
receive the same observed scores, even if they are known to differ with respect to relevant skills. 

For this reason, in educational and psychological measurement we try to craft test questions 
that match the ability of the targeted population of test takers, and quantify differences among 
test takers by eliciting responses that differentiate between higher and lower skilled respondents. 

A series of tasks that matches the skills of test takers will likely result in some correct and some 
incorrect responses. Mathematically, the variability of such a binary response (choice of correct 
versus incorrect option) is maximized when there is a 50% chance to get the tasks right. This 50/50 
criterion leads to different requirements for different test takers. More proficient test takers require 
more challenging questions in order to have (only) a 50% chance, while less proficient test takers 
require a series of easier tasks to arrive at a 50% chance of correct responses. In order to achieve this 
optimal match for all test takers, it would be necessary to adjust the test difficulty for each individual 
respondent. However, since this is only possible if the exact difficulty of all items is known (or can be 
estimated well with little error), many testing programs instead rely on a variation of this individual 
level adaptivity and adapt their tests according to the known, or estimated, average ability levels of 
pre-defined groups, rather than individuals. 
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Existing Approaches 


Country level adaptive assessment designs target particular booklets to specific populations in order 
to match ability distributions with the distribution of booklets. There are various approaches and 
assessment designs that adapt the assignment of tests to differences in target populations with respect 
to the distribution of skills, which may be estimated by previous routing instruments or inferred from 
variables such as age or educational attainment. The following section describes major approaches for 
adapting the difficulty of tests to the ability of the test taking populations. 


Starting Rules and Discontinue Rules 


In intelligence testing for individuals, for adult as well as child and adolescent populations, it is 
common to design tests that present items in the order of increasing difficulty (e.g., the Stanford 
Binet Intelligence Scale'). When first applying these tests to different age groups, it was soon noticed 
that the first few questions were not much of a challenge for older test takers, as they would get 

the first handful or so questions right in almost all cases. This led test administrators to skip these 
first few very easy items as they ’knew’ (i.e., made an inference on cases observed so far) that older 
test takers would get these easy questions right. Along the same lines, it also became apparent 

that for younger test takers, there was a point in these tests where the remaining, harder questions 
were almost impossible to solve. This in turn led test administrators to stop presenting items that 
experience had shown to be too difficult. 

Many tests of this type have a rule about how many items a test taker must get wrong 
consecutively before the testing session can be terminated. This number varies typically from 23 
items for short forms to 5-6 items for longer IQ tests. It can be shown that the discontinued items 
(those for which no response was recorded after a pre-determined number of consecutive wrong 
responses) are missing data that is ignorable’ and that the data only on what students actually took is 
sufficient to estimate ability. 


Multistage Adaptive Testing 


Multistage adaptive testing (MST) has been used in large scale international studies for adult 
populations? and can be understood as a flexible approach to assign test takers to a fixed number 
of test forms while aiming for a good, if not perfect, match between respondent ability and test 
difficulty.4 In multistage adaptive testing, the completely randomized assignment of blocks to test 
takers (the previous practice in TIMSS, PIRLS, and PISA) is modified to take into account the 
performance of the test taker on a previous block, as well as the relative difficulty of the blocks 
contained in the test design. 

In the beginning of the assessment, some form of preliminary ability estimate is required for 
each test taker so that he or she can be assigned item blocks that match their expected performance. 
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The assignment can be done deterministically, based on fixed cut-off scores, or probabilistically, 
based on a preliminary estimate of the student's ability distribution. Choosing the next block 
probabilistically ensures that at least some easy, medium and hard blocks are likely to be still available 
to all respondents at subsequent stages of the test. It also allows the assignment probability to be 
adjusted at each stage so that weak performance on a block is more likely to result in an easy block 
being assigned next, while it is still possible, with lower probability, to be presented with a medium 
or even hard block of items. Along the same lines, following strong performance on earlier blocks 
the probability of being administered a hard block of items increases, while the probability of easier 
blocks decreases.° 

The drawback of most multistage adaptive designs is that the initial starting point is either 
not adaptive because nothing is known about test takers, or it requires an initial routing block that 
produces a very rough first estimate of proficiency based on a short block of items. This estimate 
is somewhat error prone, particularly in assessments administered to a wide and diverse set of 
populations, as it assumes that the item characteristics of the routing items are known without 
error. An alternative to this approach is to use prior information based on background data such as 
education and occupation or other socio-economic data.° 


Adaptive Longitudinal Designs 


Another example of how tests are adapted to different group level ability distributions is a design that 
is being used in longitudinal large scale skill surveys.’ These designs use information on how test 
takers performed in prior assessment cycles to adaptively assign a more difficult test form to students 
who belong to a high performing group, and an easier test form to students who belong to a low 
performing group. These assessments are often 2 years apart,* so that the adaptation in this case uses 
information that dates back years in time. This approach turns out to be efficient as the performance 
at the group level is a reliable predictor of group performance at the next time point. 

Pohl? describes these designs in more detail and discusses applications in multi-cohort 
longitudinal studies of student populations. Each assessment cycle determines which test form 
should be administered to which group based on information from prior data collections. Group 
membership is based on prior performance, which may itself have been estimated using a harder or 
easier form. Over assessment cycles this provides a sequence of test forms that are tailored to decrease 
the error of measurement in proficiency estimation. It does this by increasing the expected response 
variance by matching prior performance to test forms that elicit optimal levels of systematic, ability 
related responses variability in groups of test takers. 


TIMSS & PIRLS 
ZF IEA International Study Center 
Lynch School of Education 
BOSTON COLLEGE PIRLS 2021 ASSESSMENT DESIGN 71 


—ag—$$$@$@_ 


Group Adaptive Assessment in PIRLS 2021 


Group adaptive assessment in PIRLS 2021 is implemented by dividing its 18 passages into three levels 
of passage difficulty — difficult, medium, and easy - and combining these into two levels of booklet 
difficulty: 


e More difficult booklets (9) composed of difficult or medium and difficult passages 


e Less difficult booklets (9) composed of easy or easy and medium difficult passages 


In this approach, all countries administer all 18 passages, but in varying proportions. Higher 
performing countries will administer proportionally more of the more difficult booklets while lower 
performing countries will administer proportionally more of the less difficult booklets. The goal is a 
better match between assessment difficulty and student achievement in each country. 

The group adaptive design in PIRLS 2021 involves changing from the procedure used in 
previous PIRLS cycles where booklets were randomly assigned to students at the same rate in each 
country to one where more or less difficult booklets are assigned at different rates in different 
countries. This change is intended to improve the accuracy of measurement in countries participating 
in PIRLS and provide some practical and operational advantages. More specifically, the PIRLS group 
adaptive design provides the following: 

1. Better measurement at all achievement levels by matching booklet difficulty to student ability at 
the country level 


2. All countries participate in the same assessment, maintaining full coverage of the reading 
construct while providing adaptivity at the population level 


3. Minimal disruption of the PIRLS design as there is no need for a routing block under this 
approach 


4. Improved student response rates, more student engagement, and less student frustration as 
passages are better aligned with target populations 


5. Possibility of targeting subpopulations — although the PIRLS 2021 group adaptive design is 
intended to be implemented at the country level, it also could be implemented within countries 
that have clearly defined subpopulations that vary in student ability 


As outlined in this paper there are ample examples of group-level adaptive approaches, from 
simple start/discontinue rules to elaborate longitudinal stage-based assessment designs. All these are 
based on group-level adaptivity that identifies groups of test takers which are to be assigned targeted 
test forms which are better aligned with the expected performance compared to complete random 
assignment or the use of only a single form. 


so, TIMSS & PIRLS 
Zy IEA International Study Center 


ww Lynch School of Education 
BOSTON COLLEGE PIRLS 2021 ASSESSMENT DESIGN 72 


Fg @$a >> 


The PIRLS group-adaptive design should benefit both high and low performing countries, 
in that students will be administered items that are either too difficult or too easy at a lower rate 
than in previous assessments. This improved targeting of the ability distributions will lead to more 
accurate measurement and will, as an intended side effect, likely also reduce item level non-response 
associated with administering too challenging or too easy items. Together, this is expected to lead to 
an overall improved database for reporting and secondary analyses. 
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